Re: Serious Proposal to add AccuTechnology(tm) to SpamAssassin (SA)

[Shelby Moore <co...@earthlink.net>]

Hi Sidney,

I should probably sit back and collect feedback for a while before responding.

However, you go directly to a critical point about integration, which I should probably address immediately, so as people can reflect this response (adjustment to your point) in their upcoming feedback.

BTW, I really appreciate your frank and encouraging feedback, albeit with your reservations which I will attempt to address below.


Sidney Markowitz wrote:
>You are also free to submit code for us to consider. You are free to
>attempt to convince anyone else, committer or otherwise, to write code
>and submit it.


We want to gauge the likely vote, so as to not waste time submitting code that would not be included in the distribution.


>The SpamAssassin plugin facility should make it possible to write a
>client plugin for your servers that could be distributed separately from
>SpamAssassin if the team did not want to make it part of the Apache
>distribution.


Agreed we can create our own private distribution, and we will probably do it eventually, but it may not have the same priority as if we can get included in the main distribution.

For one thing, if our distribution ended up competing in popularity with the main one, that would not be a good thing for you.  It would dilute the trademark, the focus on spamassassin.org, etc..  I do not want to encourage a fork in SpamAssassin.  On a technical side, the ways of integrating the two (Bayesian and AccuTechnology) far outstrips the integration with Razor and DCC.  Many contributors might want to experiment and improve.

Also I need to re-read the license again, to see to what degree we can promote our distribution as being "SpamAssassin + AccuTechnology".  In other words, to what degree we can make the public aware that what we are distributing is a "modified SpamAssassin".  I remember the license talks about use of trademarks.

Any way, on to your main reservation...


>That said, I would be inclined to vote -1 (veto) any submission of a
>client module that sends the entire mail message to an outside company's
>server. My reasons are 1) network overhead; 2) security; 3) privacy. The
>only one of those that I see a possible technical fix for is #2, which
>perhaps can be solved with encryption. I'm afraid that your assurances
>that you and your company and everyone with admin access to your servers
>are good people who do a great job of securing the servers from bad
>people, while probably true, are not enough.


Your concerns are appreciated and understood.

I can agree that you wouldn't want to tie the reputation of "SpamAssassin" trademark to the assurances of a 3rd party relative to network overhead, security, and privacy.

Let me address each of these concerns with clarification and alternative proposals:

1. Network Overhead

The proposed API would only send data (let's read #3 below to make it hashed data) from every Nth message.  Thus the network overhead is not more than 1/N of the server's current network overhead.  With choice of N = 10, then this should be insignificant.  See #3 for more possible reductions in overhead.

2. Security

This is partially address by not sending all (#1 above) and ways of hashing (#3 below).  As you say, it can be 100% address by doing an SSL network connection.  I suggest it will be an optional setting, since SSL has overhead and most email is sent over insecure networks any way.

3. Privacy

This is your main critical point which I agree with, if it is automatically turned ON in the distribution.

There are potential solutions.

A) First how about the possibility of setting it to OFF by default in the main distribution?  Isn't Privacy a decision best left to each server admin?  It seems to me that individual email users never have 100% privacy relative to current SpamAssassin.  The server admin controls these things, and that will still be the case with our integration.  If you force server admin to manually turn it ON, then server admin is responsible for the privacy decisions, just as he/she is with configuring the outgoing email network and intra-network issues for multiple mail accounts.  With it OFF by default, it seems to me to be a bit arbitrary or pedantic to draw a line (for the server admin) where none currently exists any way?

B) Or we can do following things, but these will have elements of our soon-to-be patented methods, so we would have to explore how our rights to charge for licenses would not be voilated by such following contributions.  I think it can be done, it will just require more careful study of the Apache license and possibly an Apache statement regarding our contribution.

However, these things reduce the ability of us to improve our algorithm in real-time (have to re-distribute code to all servers)

a) Send only the tokenized version of message. This can exceed the data size of the message, so it impacts #1. And it does not prevent the message from significantly being recovered.

b) A minature version of the centralized database is kept locally and token comparisons are done locally.  Then only each Nth tokenized version of message is sent over network.

c) The entire centralized database is kept locally.  This will drastically reduce performance unless local server has 1000s of mail accounts.  The would not be an option for small organizations, which is precisely who we are trying to give this for FREE to.


>If you solve those problems, finding some way for the client to do all
>the processing that requires access to the raw messages,


See the logic above please.


> and get your
>servers running and producing results,


The are already.   We just have to implement the API in SpamAssassin.


> you would be in the same league
>as Razor and DCC, which send only hashes over the network.


But obviously sending a hash can not find the optimum patterns in the message.  So either you've got to put the central database on each local server (#3 above) or you've got to send tokens across the network.

There is no way to get around that tradeoff.


> I see no
>reason not to have another plugin to support yet another service like
>Razor and DCC, especially if you can demonstrate better results.


I am confident we are already demonstrating much better results than Razor and DCC and with only 230 user sample!

Imagine what happens when we scale to their usership!

Thanks for the support with your reservations.

Best Regards,
Shelby Moore III