You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@spamassassin.apache.org on 2022/05/18 12:46:44 UTC
[Bug 7995] New: Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Bug ID: 7995
Summary: Add more lookups to properly leverage all Spamhaus DQS
potential (and a note about DBL)
Product: Spamassassin
Version: 4.0.0
Hardware: All
OS: All
Status: NEW
Severity: enhancement
Priority: P2
Component: spamassassin
Assignee: dev@spamassassin.apache.org
Reporter: riccardo.alfieri@spamteq.com
Target Milestone: Undefined
Good morning,
I was happy to read bug 7993 and thanks to Michael Storz for porting our
inefficient plugin feature to the core SpamAssassin 4.x codebase. I'm not
kidding, I'm happy that someone more knowledgeable than me is rewriting some of
our functions!
I'm not really in the loop with all the new SA 4.x features, so I'm going to
describe here all the lookups we do for DQS hoping that someone will find a way
to port them in the official codebase.
Good morning,
I was happy to read bug 7993 and thanks to Michael Storz for porting and
improving our inefficient plugin code to the core SpamAssassin 4.x codebase.
I'm not kidding, I'm happy that someone more knowledgeable than me is rewriting
some of our functions!
I'm not really in the loop with all the new SA 4.x features (sorry!), so I'm
going to describe here all the lookups we do for DQS hoping that someone will
find a way to port them in the official codebase. If those lookups are already
possible with 4.x (or even 3.4.6!) I also hope someone will maybe show an
example on how to do it correctly.
Here follows a breakdown of the custom functions we use in our plugin
Function: check_sh_helo.
Explanation: This function checks the domain used in the HELO/EHLO string
against DBL and ZRD.
Example: header SH_HELO_DBL eval:check_sh_helo('<key>.dbl.dq.spamhaus.net',
'^127\.0\.1\.[2-6]$')
The regex at the end is the one that is ran against the answer and defines when
to trigger a positive match
Function: check_sh_bodyemail.
Explanation: This function scans the email body looking for email addresses.
For all email addresses found, it extracts the domain and check it against DBL
and ZRD.
Example: body SH_DBL_BODY eval:check_sh_bodyemail('<key>.dbl.dq.spamhaus.net',
'^127\.0\.1\.[2-6]$')
The regex at the end is the one that is ran against the answer and defines when
to trigger a positive match
Function: check_sh_reverse
Explanation: This function checks the reverse DNS (rDNS) of the last untrusted
relay in both DBL and ZRD
Example: header SH_REVERSE_DBL
eval:check_sh_reverse('<key>.dbl.dq.spamhaus.net', '^127\.0\.1\.[2-6]$')
The regex at the end is the one that is ran against the answer and defines when
to trigger a positive match
Function: check_sh_bodyuri_a
Explanation: This function scans the email body and looks for URLs; when one is
found the hostname is then resolved, and the resulting IP address is checked in
SBL and CSS.
Example: body SH_BODYURI_REVERSE_SBL
eval:check_sh_bodyuri_a('<key>.zen.dq.spamhaus.net', '^127\.0\.0\.2$')
The regex at the end is the one that is ran against the answer and defines when
to trigger a positive match
Function: check_sh_hostname
Explanation: This function extracts whole hostnames starting from URLs in the
email body and is used to check them in the abused-legit component of DBL
Example: body SH_DBL_ABUSED_FULLHOST
eval:check_sh_hostname('<key>.dbl.dq.spamhaus.net', '^127\.0\.1\.10[2-6]$')
The regex at the end is the one that is ran against the answer and defines when
to trigger a positive match
Function: check_sh_crypto
Explanation: This functions looks for cryptowallets in the email body and
checks them in HBL.
Example: body SH_HBL_CW_BTC
eval:check_sh_crypto('_cw.<key>.hbl.dq.spamhaus.net.', '^127\.0\.3\.20$',
'\b(?:bc1|[13])[a-zA-HJ-NP-Z0-9]{25,39}\b', 'BTC')
The two regexes are, respectively, the one that looks at the DNS responses and
the one that matches the particular crypto we are looking for. In this case
it's Bitcoin, and the last parameter is the name of the crypto we want in the
logs.
I think some of those could be replaced by AskDNS, but I'm not really sure how.
Lastly, about DBL lookups. For all intent and purposes, all DBL lookups should
be done at the full hostname level instead of the domain level. all
Spam/Phish/Malware domains are wildcarded so, ie, a lookup for
"www.spamsite.com" or "www2.spamsite.com" will always return a positive match,
as the whole *.spamsite.com domain would be listed. Otherwise, abused-legit
websites are listed at a hostname level, so, ie "subsite1.wordpress.com" could
be listed, while "subsite2.wordpress.com" could be not.
I'm available either here or offlist for any clarification needed. Our final
goal is to make SpamAssassin work in the more efficient way possible with our
data, ensuring the best possible protection for all SpamAssassin users.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #16 from Henrik Krohns <ap...@hege.li> ---
(In reply to Riccardo Alfieri from comment #15)
> Well, two things. One is that we operate on scale, and distributing lists to
> hundreds of mirrors in real time is a bandwidth and time consuming process.
> If we can shed a few seconds here and there by keeping the lists as
> optimized as possible this is what we are going to do. Second, let's say we
> observe a dropbox like "user@gmail.com" , we would need to store hashes for:
>
> u.s.e.r@gmail.com
> us.e.r@gmail.com
> u.se.r@gmail.com
> use.r@gmail.com
> u.s.er@gmail.com
> us.er@gmail.com
> u.ser@gmail.com
> user@gmail.com
>
> and then do the same for @googlemail.com
>
> As you can see this approach doesn't scale
Obviously I was not suggesting that it would be feasible to start randomly
"dotting" around. All implementations already can remove the dot. The issue was
missing domain alias for HashBL, which could easily be handled on the list
side.
But as it's rspamd feature too, let's try to get it implemented the same. Just
need a short moment to think if it would be an email only option, or maybe even
a generic rewrite for any kind of found string.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #12 from Henrik Krohns <ap...@hege.li> ---
(In reply to Riccardo Alfieri from comment #11)
> Unfortunately we only list @gmail.com addresses and I think EBL do the same.
"To smoothe implementation, the EBL scripts also create hashes of email
addresses exactly as seen in the spam emails that they observe."
> It's not elegant to basically duplicate all @gmail.com listings while a
> normalization could be done on the lookup side
You can't be sure that everyone uses the same normalization code or newest
version of the code. So in my mind it's far more elegant to just support all
possible variants on the list side. Even original addresses without dots
removed etc.
> I personally +1 your suggestion about "hashbl_email_domain_alias" as it
> would be the most efficient solution
If I have time, I'll think about it.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #3 from Henrik Krohns <ap...@hege.li> ---
Quick status check on current trunk:
check_sh_helo -- check_hashbl_tag LASTEXTERNALHELO
check_sh_reverse -- check_hashbl_tag LASTEXTERNALDNS
check_sh_bodyemail -- check_hashbl_emails host option
check_sh_bodyemail_ns -- TODO
check_sh_headers -- check_hashbl_emails host option
check_sh_headers_ns -- TODO
check_sh_hostname -- URIDNSBL already does this?
check_sh_bodyuri_a -- URIDNSBL already does this?
check_sh_bodyuri_ns -- URIDNSBL already does this?
check_sh_crypto -- check_hashbl_bodyre
check_sh_attachment -- check_hashbl_attachments
check_sh_emails -- check_hashbl_emails + acl/notag/nodot for gmail stuff
Please correct me if I got something wrong.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Kevin A. McGrail <km...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #24 from Kevin A. McGrail <km...@apache.org> ---
Ricardo, it sounds like the hashbl functionality has included all the features
you need for DQS which closes this bug.
Regarding your later question, I would bring that to the user@ list rather than
a bug since you aren't sure it's a bug. One note looking at KAM.cf is that
other BTC lists use raw so I don't know why you are sha1 for a crypto wallet.
In KAM.cf, you might look at BTC_HASHBL_BLACK and see if that helps.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #10 from Henrik Krohns <ap...@hege.li> ---
(In reply to Riccardo Alfieri from comment #9)
>
> On our plugin there is a simple normalization done with this one liner:
>
> $email =~ s/\@googlemail\.com/\@gmail\.com/;
>
> You may want to include that in HashBL.pm before shipping out the pre2
I don't feel that's something that should be hardcoded in a plugin, as it
should be very trivial for you to add both variants directly in your DQS list
instead.
If there were more use cases, I might entertain some
"hashbl_email_domain_alias" config, but it seems a waste for a single entity.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #17 from Henrik Krohns <ap...@hege.li> ---
hashbl_email_domain_alias added, couldn't think of anything needing a more
generic solution.
Committed revision 1901419.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #22 from Riccardo Alfieri <ri...@spamteq.com> ---
I managed to port 99% of the functions from our 3.4.x plugin, but I'm
encountering some issues with the extraction of cryptowallets addresses.
I have a test email with the following BTC wallet in it:
1Gx3ZjJaHkXquhPzwYSFbVz1uSf[-REMOVEME-]dMGJY48
I then have a rule defined like this:
body SH_HBL_CW_BTC
eval:check_hashbl_bodyre('_cw.<key>.hbl.dq.spamhaus.net',
'sha1/max=10/shuffle', '\b(?:bc1|[13])[a-zA-HJ-NP-Z0-9]{25,39}\b',
'^127\.0\.3\.20$')
priority SH_HBL_CW_BTC -100
describe SH_HBL_CW_BTC Malicious BTC address
The resulting SHA1 hash for that BTC address is
62a692ded19caec0194c33a289bc9392de4714f0 and it's correctly listed:
$ dig +short
62a692ded19caec0194c33a289bc9392de4714f0._cw.<key>.hbl.dq.spamhaus.net
127.0.3.20
Putting the regex and the BTC on www.regexr.com gives me a positive match,
making me think that the regex is correct, however when scanning the email, SA
reports:
dbg: HashBL: SH_HBL_CW_BTC: no matches found
I guess I'm either missing something obvious or there is a bug somewhere
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #8 from Henrik Krohns <ap...@hege.li> ---
(In reply to Kevin A. McGrail from comment #7)
> Ricardo, can you also take a look at
> https://cwiki.apache.org/confluence/display/spamassassin/
> DnsBlocklistsInclusionPolicy to make sure all is good for DQS to be
> considered for 4.0 as well?
No one has suggested adding DQS to stock rules, it's not even possible since
it's subscription service.
Just making easier for people to use it without external plugins.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #11 from Riccardo Alfieri <ri...@spamteq.com> ---
Unfortunately we only list @gmail.com addresses and I think EBL do the same.
It's not elegant to basically duplicate all @gmail.com listings while a
normalization could be done on the lookup side
I personally +1 your suggestion about "hashbl_email_domain_alias" as it would
be the most efficient solution
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #5 from Henrik Krohns <ap...@hege.li> ---
(In reply to Riccardo Alfieri from comment #4)
>
> About these three functions:
>
> check_sh_hostname -- URIDNSBL already does this?
> check_sh_bodyuri_a -- URIDNSBL already does this?
> check_sh_bodyuri_ns -- URIDNSBL already does this?
>
> How would URIDNSBL accomplish those tasks? Are there some new flags to use
> in version 4.x ?
"This function extracts whole hostnames starting from URLs in the email body"
... isn't this exactly what normal "urirhsbl" rules do? The "whole hostname"
part can be done with tflags notrim (Bug 7835). Tflags "a" and "ns" already
existed since atleast 3.4.3.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #21 from Riccardo Alfieri <ri...@spamteq.com> ---
(In reply to Sidney Markowitz from comment #20)
> I haven't been following the rest of this issue, but noticed this comment.
> FYI, we released rc2 about a day ago.
Thanks I noticed and I'm now running RC2
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #14 from Henrik Krohns <ap...@hege.li> ---
(In reply to Riccardo Alfieri from comment #13)
> That is exactly what we are doing by normalizing addresses and shipping a
> plugin that normalize the query on the lookup side.
That's assuming SpamAssassin is used. Does something like rspamd support the
same things? Maybe someone does custom queries with mimedefang or something.
Even your plugin can get out of date, how do you make sure that people update
it? Same problem with SA, some people still use 5 year old versions.
> You can't really expect
> that for each gmail.com address we list we also create hashes for every
> possible variation of that same address :)
Why wouldn't I expect? Having a million or two million hashes should make no
meaningful difference to rbldnsd resource usage, or whatever you use to serve
the lists.
> The final goal is to list bad addresses. Since we know that gmail strips all
> the dots and it has also the domain googlemail.com aliased, the hashes for
> gmail addresses are created *after* the normalization. We do store the
> "original" address that triggered the listing along with other data but only
> for intel purposes.
As you have the original list, it's trivial to create the additional hashes.
Can you clarify what actual disadvantages this would have?
> I think it's SpamAssassin that needs to take care of these edge cases, not
> list mantainers.
I'm still entertaining the "alias" function, when I can figure out a logical
name and implementation for it. It will also have same problem of getting out
of date, if your users don't update things. Granted, it's likely not a
real-world issue if gmail is the only thing requiring it.
> And just to be clear, this is not a criticism but only a constructive
> suggestion from a list mantainer.
Not taking it as criticism (not that it would make any difference to me), and
I'm trying to give nothing but constructive suggestions back, as we all should.
:-)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #6 from Riccardo Alfieri <ri...@spamteq.com> ---
Ah, excellent. I'll start testing 4.x as soon as possible
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #9 from Riccardo Alfieri <ri...@spamteq.com> ---
About this particular rule:
check_sh_emails -- check_hashbl_emails + acl/notag/nodot for gmail stuff
I looked at the code and apparently there is no googlemail.com->gmail.com
normalization.
As you probably know you can use @googlemail.com intead of @gmail.com and have
the email delivered to the same mailbox in both cases. Spammers sometimes use
@googlemail.com addresses to try to evade listings.
On our plugin there is a simple normalization done with this one liner:
$email =~ s/\@googlemail\.com/\@gmail\.com/;
You may want to include that in HashBL.pm before shipping out the pre2
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Riccardo Alfieri <ri...@spamteq.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |riccardo.alfieri@spamteq.co
| |m
--- Comment #2 from Riccardo Alfieri <ri...@spamteq.com> ---
Yes, I intentionally left it out as I saw that new function that you committed.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Sidney Markowitz <si...@sidney.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sidney@sidney.com
--- Comment #20 from Sidney Markowitz <si...@sidney.com> ---
(In reply to Riccardo Alfieri from comment #19)
> I've just started porting all our lookups to SA's RC1.
I haven't been following the rest of this issue, but noticed this comment. FYI,
we released rc2 about a day ago.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #19 from Riccardo Alfieri <ri...@spamteq.com> ---
(In reply to Kevin A. McGrail from comment #18)
> Can we close this bug?
>
> Ricardo, if you want to suggest DQS for default rule inclusion, happy to
> discuss that as well.
Hi Kevin,
apologies for the delay, your email got lost in the inbox :)
I've just started porting all our lookups to SA's RC1. I'll update this bug in
the near future with a feedback
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Kevin A. McGrail <km...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@apache.org
--- Comment #7 from Kevin A. McGrail <km...@apache.org> ---
(In reply to Riccardo Alfieri from comment #6)
> Ah, excellent. I'll start testing 4.x as soon as possible
Ricardo, can you also take a look at
https://cwiki.apache.org/confluence/display/spamassassin/DnsBlocklistsInclusionPolicy
to make sure all is good for DQS to be considered for 4.0 as well?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #4 from Riccardo Alfieri <ri...@spamteq.com> ---
I think you can safely postpone the TODO as those two functions are there but
there is no dataset to query (SBL is not 100% correct for this use case, we may
publish a dedicated zone in the future)
About these three functions:
check_sh_hostname -- URIDNSBL already does this?
check_sh_bodyuri_a -- URIDNSBL already does this?
check_sh_bodyuri_ns -- URIDNSBL already does this?
How would URIDNSBL accomplish those tasks? Are there some new flags to use in
version 4.x ?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #13 from Riccardo Alfieri <ri...@spamteq.com> ---
(In reply to Henrik Krohns from comment #12)
>
> You can't be sure that everyone uses the same normalization code or newest
> version of the code. So in my mind it's far more elegant to just support all
> possible variants on the list side. Even original addresses without dots
> removed etc.
That is exactly what we are doing by normalizing addresses and shipping a
plugin that normalize the query on the lookup side. You can't really expect
that for each gmail.com address we list we also create hashes for every
possible variation of that same address :)
The final goal is to list bad addresses. Since we know that gmail strips all
the dots and it has also the domain googlemail.com aliased, the hashes for
gmail addresses are created *after* the normalization. We do store the
"original" address that triggered the listing along with other data but only
for intel purposes.
And I can assure you that we have *plenty* of examples of spammers trying to
game filters by adding random dots and changing the domain for their dropboxes.
I think it's SpamAssassin that needs to take care of these edge cases, not list
mantainers. And just to be clear, this is not a criticism but only a
constructive suggestion from a list mantainer. We love SpamAssassin and would
love for it to work in the most efficient way with our data. In the end you
already have "nodot" and "notag" options, adding another option for aliased
domain would probably render SA 4.x more "future proof".
And if this is not a feature that will be included in 4.x then we'll need to
write another plugin, or ship a dedicated HashBL.pm but this is a suboptimal
solution that many people will have some difficulties implementing, resulting
in more spam reaching the recipient (that is what we all are trying to avoid)
>
> If I have time, I'll think about it.
If I had the necessary coding abilities I would help you, but unfortunately my
coding style is vastly inferior compared to yours :(
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #18 from Kevin A. McGrail <km...@apache.org> ---
Can we close this bug?
Ricardo, if you want to suggest DQS for default rule inclusion, happy to
discuss that as well.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #15 from Riccardo Alfieri <ri...@spamteq.com> ---
(In reply to Henrik Krohns from comment #14)
> (In reply to Riccardo Alfieri from comment #13)
> > That is exactly what we are doing by normalizing addresses and shipping a
> > plugin that normalize the query on the lookup side.
>
> That's assuming SpamAssassin is used. Does something like rspamd support the
> same things? Maybe someone does custom queries with mimedefang or something.
>
Yes, Rspamd does exactly those normalizations, see the functions
"check_gmail_user" and "check_googlemail" in
https://github.com/rspamd/rspamd/blob/master/lualib/lua_util.lua
It does it hardcoded, maybe because gmail is the only provider (that I know of)
that do such weird things with his mailboxes
> Even your plugin can get out of date, how do you make sure that people
> update it? Same problem with SA, some people still use 5 year old versions.
>
That is why we ensured that our plugin works with SA 3.4.1+, because many
distributions still package that version. We encourage our customers to keep an
eye for updates and try the latest version before reporting a bug.
> Why wouldn't I expect? Having a million or two million hashes should make no
> meaningful difference to rbldnsd resource usage, or whatever you use to
> serve the lists.
Well, two things. One is that we operate on scale, and distributing lists to
hundreds of mirrors in real time is a bandwidth and time consuming process. If
we can shed a few seconds here and there by keeping the lists as optimized as
possible this is what we are going to do. Second, let's say we observe a
dropbox like "user@gmail.com" , we would need to store hashes for:
u.s.e.r@gmail.com
us.e.r@gmail.com
u.se.r@gmail.com
use.r@gmail.com
u.s.er@gmail.com
us.er@gmail.com
u.ser@gmail.com
user@gmail.com
and then do the same for @googlemail.com
As you can see this approach doesn't scale
>
> Not taking it as criticism (not that it would make any difference to me),
> and I'm trying to give nothing but constructive suggestions back, as we all
> should. :-)
Of course! I just wanted to explicitly say that to avoid giving the impression
of wanting to "bully" you in writing a function only for our convenience
I hope we'll reach a consensus on this :)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
Henrik Krohns <ap...@hege.li> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |apache@hege.li
--- Comment #1 from Henrik Krohns <ap...@hege.li> ---
I think you left out check_sh_attachment? Just committed our
check_hashbl_attachments function.. will look what is left to do.. helo and
reverse can probably be done easily with check_hashbl_tag
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #25 from Riccardo Alfieri <ri...@spamteq.com> ---
Thanks Kevin, this is just to confirm that everything finally works as
expected.
For posterity, my error was to not use a capture group in the regex included in
the check_hashbl_bodyre function. Fixing that and adding "case" to the rule
option fixed everything
Again, thanks everyone for your work!
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7995] Add more lookups to properly leverage all Spamhaus DQS potential (and a note about DBL)
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7995
--- Comment #23 from Riccardo Alfieri <ri...@spamteq.com> ---
Would it be better if I move the previous report to a new bug report?
--
You are receiving this mail because:
You are the assignee for the bug.