You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Sebastian Arcus <s....@open-t.co.uk> on 2015/12/02 12:51:41 UTC
Is it worth transferring bayes data between different sites?
I hope I'm not exceeding the patience of the list by posting a third
question in two days :-)
I realise the above question is a "soft" question, probably without a
definite "yes" or "no" answer. I am hoping that people with experience
of using SA in various environments might be able to throw in some
opinions. Based on the documentation, it is clearly possible to transfer
a bayes database from one install to another - specially if it is a
sitewide database. What I was wondering is if it is worth doing so from
a results point of view. For example, I now have a nicely trained bayes
with a few thousands of my own ham, clean, hand-picked emails and a few
hundred spam emails. Would there be a significant benefit in taking this
data and using it to setup a fresh SA install on a client's server? Or
the fact that my specific email usage pattern and content of the email I
receive being different from the one of another organisation would
render the bayes tokens/data useless - and I am better off starting from
scratch there?
I've tried searching online for a discussion on this topic, but the only
relevant bit I found is this unanswered post from 2004:
https://mail-archives.apache.org/mod_mbox/spamassassin-users/200406.mbox/%3C20040621150409.GA23159@publinet.it%3E
Re: Is it worth transferring bayes data between different sites?
Posted by Sebastian Arcus <s....@open-t.co.uk>.
On 03/12/15 00:29, Charles Sprickman wrote:
> Reindl Harald <h....@thelounge.net> wrote:
>
>>
>> Am 02.12.2015 um 21:50 schrieb Charles Sprickman:
>>> Reindl Harald <h....@thelounge.net> wrote:
>>>
>>>> Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
>>>>> I hope I'm not exceeding the patience of the list by posting a third
>>>>> question in two days :-)
>>>>>
>>>>> I realise the above question is a "soft" question, probably without a
>>>>> definite "yes" or "no" answer. I am hoping that people with experience
>>>>> of using SA in various environments might be able to throw in some
>>>>> opinions. Based on the documentation, it is clearly possible to transfer
>>>>> a bayes database from one install to another - specially if it is a
>>>>> sitewide database. What I was wondering is if it is worth doing so from
>>>>> a results point of view
>>>> we use our global bayes on the incoming MX and share it with our submission servers to stop outgoing spam from hacked accounts
>>> This is a bit OT, but I have had a hard time finding how to setup a global bayes DB rather than having everything done on a per-user basis. Looking around the SA wiki, I don’t see global DBs addressed. Any tips?
>> https://wiki.apache.org/spamassassin/SiteWideBayesSetup
>>
>> in case you are runnign spamass-milter that's even the logical default because your milter is running as it's own user, with it's own .spamassassin directory in the userhome which contains the db
> I had a look at that page - I use mysql to store the data, have multiple spamd boxes, and spamc on the inbound servers passing mail to spamd once all the “front door” checks are done. In that config, I end up with unique per-user bayes tokens. I’m looking to just pool everyone together, but don’t see an obvious way to do that. It seems like folks in this thread are however doing that somehow (perhaps just because they are using a milter or similar).
In case in helps: I use SA with exim - and Exim talks over Unix sockets
to spamd daemon. I've used the instructions at the wiki page above to
setup the sitewide bayes database - but I don't use MySQL - and it all
seems to work as expected.
Re: Is it worth transferring bayes data between different sites?
Posted by Jari Fredriksson <ja...@iki.fi>.
On 12/03/2015 01:38 PM, Jari Fredriksson wrote:
> On 12/03/2015 02:29 AM, Charles Sprickman wrote:
>> I had a look at that page - I use mysql to store the data, have
>> multiple spamd boxes, and spamc on the inbound servers passing mail
>> to spamd once all the “front door” checks are done. In that config,
>> I end up with unique per-user bayes tokens. I’m looking to just pool
>> everyone together, but don’t see an obvious way to do that. It seems
>> like folks in this thread are however doing that somehow (perhaps
>> just because they are using a milter or similar).
>>
>> Thanks,
>>
>> Charles
>>
>>
>
> I have a similar setup. I use "spamc -d spamd -u spam ..." and I think
> that -u spam is all it takes to make it site wide. Not very complex?
>
> br. jarif
>
Btw, I really have a similar setup, as host name "spamd" points to
haproxy having multiple back ends on it for spamassassin.
br. jarif
Re: Is it worth transferring bayes data between different sites?
Posted by Kris Deugau <kd...@vianet.ca>.
Jari Fredriksson wrote:
> On 12/03/2015 02:29 AM, Charles Sprickman wrote:
>> I had a look at that page - I use mysql to store the data, have
>> multiple spamd boxes, and spamc on the inbound servers passing mail to
>> spamd once all the “front door” checks are done. In that config, I
>> end up with unique per-user bayes tokens. I’m looking to just pool
>> everyone together, but don’t see an obvious way to do that. It seems
>> like folks in this thread are however doing that somehow (perhaps just
>> because they are using a milter or similar).
> I have a similar setup. I use "spamc -d spamd -u spam ..." and I think
> that -u spam is all it takes to make it site wide. Not very complex?
That'll give you a global Bayes, but it also eliminate any per-user
settings you might want to keep using, because you're not letting spamd
"know" about any different users.
We use these settings (watch for word wrap):
---
bayes_store_module Mail::SpamAssassin::BayesStore::SQL
bayes_sql_dsn DBI:mysql:spamassassin:[ip]
bayes_sql_username spamassassin
bayes_sql_password ********
bayes_sql_override_username spamassassin
# awl
auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList
user_awl_dsn DBI:mysql:spamassassin:[ip]
user_awl_sql_username spamassassin
user_awl_sql_password ********
user_awl_sql_table awl
# userprefs. worksforme(TM)
user_scores_dsn DBI:mysql:spamassassin:[ip]
user_scores_sql_username spamassassin
user_scores_sql_password ********
# Need a custom query to do domainwide settings. Default does not have
the third WHERE clause
user_scores_sql_custom_query SELECT preference, value FROM _TABLE_
WHERE username = _USERNAME_ OR username = '@GLOBAL' OR username =
concat('@~',_DOMAIN_) ORDER BY username ASC
# don't pass mail through unscanned if there's an error trying to get
userprefs.
# Note that "no userpref entries" is NOT an error.
# Note also this requires a custom patch
user_scores_fallback_to_global 1
---
to have per-user AWL, SQL userprefs including domainwide settings, and
global Bayes all in the same MySQL database.
A read of the fine manual page (Mail::SpamAssassin::Conf) will usually
turn up all the necessary directives for whatever you're trying to do.
-kgd
Re: Is it worth transferring bayes data between different sites?
Posted by Jari Fredriksson <ja...@iki.fi>.
On 12/03/2015 02:29 AM, Charles Sprickman wrote:
> I had a look at that page - I use mysql to store the data, have multiple spamd boxes, and spamc on the inbound servers passing mail to spamd once all the “front door” checks are done. In that config, I end up with unique per-user bayes tokens. I’m looking to just pool everyone together, but don’t see an obvious way to do that. It seems like folks in this thread are however doing that somehow (perhaps just because they are using a milter or similar).
>
> Thanks,
>
> Charles
>
>
I have a similar setup. I use "spamc -d spamd -u spam ..." and I think
that -u spam is all it takes to make it site wide. Not very complex?
br. jarif
Re: Is it worth transferring bayes data between different sites?
Posted by Kris Deugau <kd...@vianet.ca>.
Charles Sprickman wrote:
> I had a look at that page - I use mysql to store the data, have multiple spamd boxes, and spamc on the inbound servers passing mail to spamd once all the “front door” checks are done. In that config, I end up with unique per-user bayes tokens. I’m looking to just pool everyone together, but don’t see an obvious way to do that. It seems like folks in this thread are however doing that somehow (perhaps just because they are using a milter or similar).
Really short answer:
bayes_sql_username spamassassin
man Mail::SpamAssassin::Conf (IIRC) for details. That directive
overrides the spamd per-user behaviour for Bayes, putting it all in one
basket.
-kgd
Re: Is it worth transferring bayes data between different sites?
Posted by Charles Sprickman <sp...@bway.net>.
Reindl Harald <h....@thelounge.net> wrote:
>
>
> Am 02.12.2015 um 21:50 schrieb Charles Sprickman:
>> Reindl Harald <h....@thelounge.net> wrote:
>>
>>> Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
>>>> I hope I'm not exceeding the patience of the list by posting a third
>>>> question in two days :-)
>>>>
>>>> I realise the above question is a "soft" question, probably without a
>>>> definite "yes" or "no" answer. I am hoping that people with experience
>>>> of using SA in various environments might be able to throw in some
>>>> opinions. Based on the documentation, it is clearly possible to transfer
>>>> a bayes database from one install to another - specially if it is a
>>>> sitewide database. What I was wondering is if it is worth doing so from
>>>> a results point of view
>>>
>>> we use our global bayes on the incoming MX and share it with our submission servers to stop outgoing spam from hacked accounts
>>
>> This is a bit OT, but I have had a hard time finding how to setup a global bayes DB rather than having everything done on a per-user basis. Looking around the SA wiki, I don’t see global DBs addressed. Any tips?
>
> https://wiki.apache.org/spamassassin/SiteWideBayesSetup
>
> in case you are runnign spamass-milter that's even the logical default because your milter is running as it's own user, with it's own .spamassassin directory in the userhome which contains the db
I had a look at that page - I use mysql to store the data, have multiple spamd boxes, and spamc on the inbound servers passing mail to spamd once all the “front door” checks are done. In that config, I end up with unique per-user bayes tokens. I’m looking to just pool everyone together, but don’t see an obvious way to do that. It seems like folks in this thread are however doing that somehow (perhaps just because they are using a milter or similar).
Thanks,
Charles
Re: Is it worth transferring bayes data between different sites?
Posted by Reindl Harald <h....@thelounge.net>.
Am 02.12.2015 um 21:50 schrieb Charles Sprickman:
> Reindl Harald <h....@thelounge.net> wrote:
>
>>
>>
>> Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
>>> I hope I'm not exceeding the patience of the list by posting a third
>>> question in two days :-)
>>>
>>> I realise the above question is a "soft" question, probably without a
>>> definite "yes" or "no" answer. I am hoping that people with experience
>>> of using SA in various environments might be able to throw in some
>>> opinions. Based on the documentation, it is clearly possible to transfer
>>> a bayes database from one install to another - specially if it is a
>>> sitewide database. What I was wondering is if it is worth doing so from
>>> a results point of view
>>
>> we use our global bayes on the incoming MX and share it with our submission servers to stop outgoing spam from hacked accounts
>
> This is a bit OT, but I have had a hard time finding how to setup a global bayes DB rather than having everything done on a per-user basis. Looking around the SA wiki, I don’t see global DBs addressed. Any tips?
https://wiki.apache.org/spamassassin/SiteWideBayesSetup
in case you are runnign spamass-milter that's even the logical default
because your milter is running as it's own user, with it's own
.spamassassin directory in the userhome which contains the db
Re: Is it worth transferring bayes data between different sites?
Posted by Charles Sprickman <sp...@bway.net>.
Reindl Harald <h....@thelounge.net> wrote:
>
>
> Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
>> I hope I'm not exceeding the patience of the list by posting a third
>> question in two days :-)
>>
>> I realise the above question is a "soft" question, probably without a
>> definite "yes" or "no" answer. I am hoping that people with experience
>> of using SA in various environments might be able to throw in some
>> opinions. Based on the documentation, it is clearly possible to transfer
>> a bayes database from one install to another - specially if it is a
>> sitewide database. What I was wondering is if it is worth doing so from
>> a results point of view
>
> we use our global bayes on the incoming MX and share it with our submission servers to stop outgoing spam from hacked accounts
This is a bit OT, but I have had a hard time finding how to setup a global bayes DB rather than having everything done on a per-user basis. Looking around the SA wiki, I don’t see global DBs addressed. Any tips?
Thanks,
Charles
>
> additionally we share our bayes with another company which pulls the dumps if the hash file is different every 30 minutes
>
> we as well as the other company does mail hosting on ISP level and the results on both sides are perfect - we share even scorings, whitelists, custom body/subject-rules and the summary is: at least in the same country sharing spamfilter configurations works like a charme
Re: Is it worth transferring bayes data between different sites?
Posted by Reindl Harald <h....@thelounge.net>.
Am 03.12.2015 um 12:41 schrieb Jeroen de Neef:
> I'd like to teach my bayes correctly especially since I don't get a lot
> of emails, thanks to Reindl's list I will ignore those headers from now on.
> But I don't want it to learn that the /*****spam*****/ in the subject
> means that it is spam or ham, is there a way I can remove it before
> throwing it at the bayesian filter? Perhaps an extra line in the config
> or a bash script?
just add a replace in the php-script i posted before it verifies the new
content against the old one to decide if the file needs to be rewritten
for such cleanups and anonymize i use seperated scripts to keep the code
clean, one of them also reads the postfix configration and replaces own
domains and email-addresses with "me@example.com"
"I will ignore those headers from now on" - the ignore configuration is
not enough, hence the formail script to strip the headers completly from
the samples
the Received header is a special case - if the samples don't have any
Received header you get *completly* different bayes results compared
with a always identical one, hence i strip them all and add a generic at
the end on top of the file
that leads also in have a dramatical reduced token number because you
have at the end only one token for Received with the same date, time, host
Re: Is it worth transferring bayes data between different sites?
Posted by Jeroen de Neef <je...@gmail.com>.
Hello all,
I'd like to teach my bayes correctly especially since I don't get a lot of
emails, thanks to Reindl's list I will ignore those headers from now on.
But I don't want it to learn that the ******spam****** in the subject means
that it is spam or ham, is there a way I can remove it before throwing it
at the bayesian filter? Perhaps an extra line in the config or a bash
script?
Kind regards,
Jeroen
2015-12-03 11:00 GMT+01:00 Reindl Harald <h....@thelounge.net>:
>
>
> Am 03.12.2015 um 10:47 schrieb Sebastian Arcus:
>
>> On 03/12/15 01:40, Reindl Harald wrote:
>>
>>>
>>>
>>> Am 03.12.2015 um 01:14 schrieb Alex:
>>>
>>>> On Wed, Dec 2, 2015 at 6:34 PM, Dave Warren <da...@hireahit.com> wrote:
>>>>
>>>>> On 2015-12-02 09:14, Sebastian Arcus wrote:
>>>>>
>>>>>>
>>>>>> Perfect - that's exactly the sort of real-life based advice I was
>>>>>> looking
>>>>>> for. Many thanks!
>>>>>>
>>>>>
>>>>> I run a small shared hosting environment, with a global bayes for
>>>>> all users
>>>>> as not enough users are ready/willing/able to take the time to sort ham
>>>>> (although more will press "this is spam") and in general, the
>>>>> results work
>>>>> out well enough.
>>>>>
>>>>
>>>> A portion of the bayes database is the header information from the
>>>> email. What does it mean for those headers that contain info specific
>>>> to a particular domain or site when it's transferred to another domain
>>>> or site where those specifics will be different?
>>>>
>>>
>>> see attached php/formail-script and list of ignored/stripped headers
>>>
>>> we strip a large portion of headers including especially the Received
>>> headers with "formail" and preprend a egenric one on top from all
>>> samples before train them
>>>
>> Does that mean that transferring bayes databases between sites without
>> stripping the headers wouldn't work - or it is just more effective if
>> one strips the headers?
>>
>
> it worked without strip them around 6 months
> but it works better now
>
> see the 77.72% BAYES_00 which would be more but some trained ham is in
> shortcircuit and so don't touch bayes at all
>
> "SPAMMY" means >= BAYES_60 in the stats
>
> BAYES_00 3914 77.72 %
> BAYES_05 87 1.72 %
> BAYES_20 134 2.66 %
> BAYES_40 108 2.14 %
> BAYES_50 288 5.71 %
> BAYES_60 61 1.21 %
> BAYES_80 45 0.89 %
> BAYES_95 34 0.67 %
> BAYES_99 365 7.24 %
> BAYES_999 319 6.33 %
>
> DELIVERED 6609 95.18 %
> DNSWL 6249 90.00 %
> SPF 4586 66.05 %
> SPF/DKIM WL 1880 27.07 %
> SHORTCIRCUIT 1900 27.36 %
>
> BLOCKED 515 7.41 %
> SPAMMY 505 7.27 % 98.05 % (OF TOTAL BLOCKED)
>
>
>
>
Re: Is it worth transferring bayes data between different sites?
Posted by Reindl Harald <h....@thelounge.net>.
Am 03.12.2015 um 10:47 schrieb Sebastian Arcus:
> On 03/12/15 01:40, Reindl Harald wrote:
>>
>>
>> Am 03.12.2015 um 01:14 schrieb Alex:
>>> On Wed, Dec 2, 2015 at 6:34 PM, Dave Warren <da...@hireahit.com> wrote:
>>>> On 2015-12-02 09:14, Sebastian Arcus wrote:
>>>>>
>>>>> Perfect - that's exactly the sort of real-life based advice I was
>>>>> looking
>>>>> for. Many thanks!
>>>>
>>>> I run a small shared hosting environment, with a global bayes for
>>>> all users
>>>> as not enough users are ready/willing/able to take the time to sort ham
>>>> (although more will press "this is spam") and in general, the
>>>> results work
>>>> out well enough.
>>>
>>> A portion of the bayes database is the header information from the
>>> email. What does it mean for those headers that contain info specific
>>> to a particular domain or site when it's transferred to another domain
>>> or site where those specifics will be different?
>>
>> see attached php/formail-script and list of ignored/stripped headers
>>
>> we strip a large portion of headers including especially the Received
>> headers with "formail" and preprend a egenric one on top from all
>> samples before train them
> Does that mean that transferring bayes databases between sites without
> stripping the headers wouldn't work - or it is just more effective if
> one strips the headers?
it worked without strip them around 6 months
but it works better now
see the 77.72% BAYES_00 which would be more but some trained ham is in
shortcircuit and so don't touch bayes at all
"SPAMMY" means >= BAYES_60 in the stats
BAYES_00 3914 77.72 %
BAYES_05 87 1.72 %
BAYES_20 134 2.66 %
BAYES_40 108 2.14 %
BAYES_50 288 5.71 %
BAYES_60 61 1.21 %
BAYES_80 45 0.89 %
BAYES_95 34 0.67 %
BAYES_99 365 7.24 %
BAYES_999 319 6.33 %
DELIVERED 6609 95.18 %
DNSWL 6249 90.00 %
SPF 4586 66.05 %
SPF/DKIM WL 1880 27.07 %
SHORTCIRCUIT 1900 27.36 %
BLOCKED 515 7.41 %
SPAMMY 505 7.27 % 98.05 % (OF TOTAL BLOCKED)
Re: Is it worth transferring bayes data between different sites?
Posted by Sebastian Arcus <s....@open-t.co.uk>.
On 03/12/15 01:40, Reindl Harald wrote:
>
>
> Am 03.12.2015 um 01:14 schrieb Alex:
>> On Wed, Dec 2, 2015 at 6:34 PM, Dave Warren <da...@hireahit.com> wrote:
>>> On 2015-12-02 09:14, Sebastian Arcus wrote:
>>>>
>>>> Perfect - that's exactly the sort of real-life based advice I was
>>>> looking
>>>> for. Many thanks!
>>>
>>> I run a small shared hosting environment, with a global bayes for
>>> all users
>>> as not enough users are ready/willing/able to take the time to sort ham
>>> (although more will press "this is spam") and in general, the
>>> results work
>>> out well enough.
>>
>> A portion of the bayes database is the header information from the
>> email. What does it mean for those headers that contain info specific
>> to a particular domain or site when it's transferred to another domain
>> or site where those specifics will be different?
>
> see attached php/formail-script and list of ignored/stripped headers
>
> we strip a large portion of headers including especially the Received
> headers with "formail" and preprend a egenric one on top from all
> samples before train them
Does that mean that transferring bayes databases between sites without
stripping the headers wouldn't work - or it is just more effective if
one strips the headers?
Re: Is it worth transferring bayes data between different sites?
Posted by Reindl Harald <h....@thelounge.net>.
Am 03.12.2015 um 01:14 schrieb Alex:
> On Wed, Dec 2, 2015 at 6:34 PM, Dave Warren <da...@hireahit.com> wrote:
>> On 2015-12-02 09:14, Sebastian Arcus wrote:
>>>
>>> Perfect - that's exactly the sort of real-life based advice I was looking
>>> for. Many thanks!
>>
>> I run a small shared hosting environment, with a global bayes for all users
>> as not enough users are ready/willing/able to take the time to sort ham
>> (although more will press "this is spam") and in general, the results work
>> out well enough.
>
> A portion of the bayes database is the header information from the
> email. What does it mean for those headers that contain info specific
> to a particular domain or site when it's transferred to another domain
> or site where those specifics will be different?
see attached php/formail-script and list of ignored/stripped headers
we strip a large portion of headers including especially the Received
headers with "formail" and preprend a egenric one on top from all
samples before train them
on one hand you need more samples for perfect results
on teh other hand you get rid of the problem of senders like Yahoo/Gmail
which are spam *and* ham what means you need a ton of ham samples from
them too to prevent bayes get a tendency
short said: we try to anonymize and strip down samples to contain mostly
body and subject informations and keep them forever
sadly SA ignores "bayes_ignore_header received" and acty weird if there
is no Received at all in case of body, on the other hand with the
attached scripts the corpus got rid of 250000 headers finally
Re: Is it worth transferring bayes data between different sites?
Posted by Alex <my...@gmail.com>.
Hi,
On Wed, Dec 2, 2015 at 6:34 PM, Dave Warren <da...@hireahit.com> wrote:
> On 2015-12-02 09:14, Sebastian Arcus wrote:
>>
>> Perfect - that's exactly the sort of real-life based advice I was looking
>> for. Many thanks!
>
> I run a small shared hosting environment, with a global bayes for all users
> as not enough users are ready/willing/able to take the time to sort ham
> (although more will press "this is spam") and in general, the results work
> out well enough.
A portion of the bayes database is the header information from the
email. What does it mean for those headers that contain info specific
to a particular domain or site when it's transferred to another domain
or site where those specifics will be different?
Thanks,
Alex
Re: Is it worth transferring bayes data between different sites?
Posted by Dave Warren <da...@hireahit.com>.
On 2015-12-02 09:14, Sebastian Arcus wrote:
> Perfect - that's exactly the sort of real-life based advice I was
> looking for. Many thanks!
I run a small shared hosting environment, with a global bayes for all
users as not enough users are ready/willing/able to take the time to
sort ham (although more will press "this is spam") and in general, the
results work out well enough.
Sharing bayes between servers or sites would not seem to be particularly
different than a shared bayes between multiple customers in a shared
hosting, as long as the "typical end user" is similar. If you have a
viagra dealer or diet pill retailer as one of your customers, your
mileage may vary and they may need more personalization, but in general,
for typical SOHO and SMB customers, spammy spam is spammy spam and
pretty widely distributed.
From what I see, it's ham that varies a lot per-user, and so while we
try to train bayes across a wide range of ham sets, we also do a lot of
automated whitelisting based on user behaviour based on mail that users
send, or mail that users keep in their mailboxes so that we can skip
spam filtering entirely for as much "wanted" mail as possible. We also
try to reduce filtering on replies based on the "In-Reply-To:" header
containing headers that match certain formats (such as what our webmail
produces, what we add to messages missing this header, and a few other
formats), so it's possible that someone else who borrowed our bayes
database might end up seeing a higher false-positive rate.
We avoid training big companies (Amazon, eBay, etc) as spam even when
they spam, as long as it's clearly identified in a blockable way,
instead providing users the ability to block senders outright when
applicable.
Sure, there are errors and mistakes, by and large, bayes works out the
details in a shared environment, a multi-server environment shouldn't be
too different, as long as the customer base is similar.
--
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren
Re: Is it worth transferring bayes data between different sites?
Posted by RW <rw...@googlemail.com>.
On Wed, 2 Dec 2015 17:14:22 +0000
Sebastian Arcus wrote:
> On 02/12/15 12:55, Reindl Harald wrote:
> >
> >
> > Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
> >> I hope I'm not exceeding the patience of the list by posting a
> >> third question in two days :-)
> >>
> >> I realise the above question is a "soft" question, probably
> >> without a definite "yes" or "no" answer.
Yery true.
> > additionally we share our bayes with another company which pulls
> > the dumps if the hash file is different every 30 minutes
> >
> > we as well as the other company does mail hosting on ISP level and
> > the results on both sides are perfect - we share even scorings,
> > whitelists, custom body/subject-rules and the summary is: at least
> > in the same country sharing spamfilter configurations works like a
> > charme
>
> Perfect - that's exactly the sort of real-life based advice I was
> looking for. Many thanks!
It's not really surprising that the diverse mail of 2 similar ISPs is
similar for Bayes, especially with the headers removed. Whether your
ham looks like your client's ham is an entirely different matter. If
the ham isn't similar then using your ham-heavy database is likely to
be sub-optimal.
There's also the ham:spam ratio - at one point you quoted a figure of
12000:300. An imbalance is not intrinsically wrong, but it could cause
problems if you transplant it into a system where new training occurs at
a very different ratio. Any new tokens that appear in the second system
are heavily skewed to being treated as spammy. What's particularly bad
is if you strip headers in your corpus and then the client goes on to
train without stripping them, then neutral tokens that got stripped
enter the database as heavily spammy.
Re: Is it worth transferring bayes data between different sites?
Posted by Sebastian Arcus <s....@open-t.co.uk>.
On 02/12/15 12:55, Reindl Harald wrote:
>
>
> Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
>> I hope I'm not exceeding the patience of the list by posting a third
>> question in two days :-)
>>
>> I realise the above question is a "soft" question, probably without a
>> definite "yes" or "no" answer. I am hoping that people with experience
>> of using SA in various environments might be able to throw in some
>> opinions. Based on the documentation, it is clearly possible to transfer
>> a bayes database from one install to another - specially if it is a
>> sitewide database. What I was wondering is if it is worth doing so from
>> a results point of view
>
> we use our global bayes on the incoming MX and share it with our
> submission servers to stop outgoing spam from hacked accounts
>
> additionally we share our bayes with another company which pulls the
> dumps if the hash file is different every 30 minutes
>
> we as well as the other company does mail hosting on ISP level and the
> results on both sides are perfect - we share even scorings,
> whitelists, custom body/subject-rules and the summary is: at least in
> the same country sharing spamfilter configurations works like a charme
Perfect - that's exactly the sort of real-life based advice I was
looking for. Many thanks!
Re: Is it worth transferring bayes data between different sites?
Posted by Reindl Harald <h....@thelounge.net>.
Am 02.12.2015 um 12:51 schrieb Sebastian Arcus:
> I hope I'm not exceeding the patience of the list by posting a third
> question in two days :-)
>
> I realise the above question is a "soft" question, probably without a
> definite "yes" or "no" answer. I am hoping that people with experience
> of using SA in various environments might be able to throw in some
> opinions. Based on the documentation, it is clearly possible to transfer
> a bayes database from one install to another - specially if it is a
> sitewide database. What I was wondering is if it is worth doing so from
> a results point of view
we use our global bayes on the incoming MX and share it with our
submission servers to stop outgoing spam from hacked accounts
additionally we share our bayes with another company which pulls the
dumps if the hash file is different every 30 minutes
we as well as the other company does mail hosting on ISP level and the
results on both sides are perfect - we share even scorings, whitelists,
custom body/subject-rules and the summary is: at least in the same
country sharing spamfilter configurations works like a charme