You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Reindl Harald <h....@thelounge.net> on 2016/07/05 12:01:17 UTC
URIBL randomly not triggered for the same message
i have here a message with URIBL_ABUSE_SURBL Contains an URL listed in
the ABUSE SURBL blocklist
50% of all tries against spamd it does NOT hit while the scantime for
the whole message is arounnd 3 seconds - since there is a local
unbound-cache with
cache-min-ttl: 300
cache-max-ttl: 10800
it's impossible that there are happening dns timeouts and i can observe
the same behavior randomly with URIBL_LOCAL where the unbound dns cache
on 127.0.0.1:53 talks to rblsdnsd on 127.0.0.1:1053
that smells why ever very unrelieable and frankly i observed similar
with SPF_PASS / SHORTCIRCUIT where people within 5 seconds get the same
message and one get USER_IN_SPF_WHITELIST while the other goes through
all tests
Re: URIBL randomly not triggered for the same message
Posted by Benny Pedersen <me...@junc.eu>.
On 2016-07-26 11:39, Reindl Harald wrote:
> sadly it don't work as expected
> https://bugzilla.redhat.com/show_bug.cgi?id=1360222
add forward-first: yes to forward zone
without you are qquery stale data in unbound
no i do not use bind9 now :=)
Re: URIBL randomly not triggered for the same message
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.07.2016 um 17:40 schrieb Reindl Harald:
> Am 06.07.2016 um 17:35 schrieb John Hardin:
>> On Wed, 6 Jul 2016, Paul Stead wrote:
>>
>>> On 06/07/16 16:16, John Hardin wrote:
>>>> Does that cache-min-ttl also affect NXDOMAIN? Is it possible to
>>>> configure different TTL for NXDOMAIN (relatively low) and positive
>>>> results (relatively high)?
>>>
>>> For this cache-max-negative-ttl exists :)
>>
>> :) It's obvious I don't use unbound...
>>
>> Reindl, does that approach help?
>
> sounds good and at leat i don't get any error by using
> unbound-1.5.8-2.fc23.x86_64 and the follwoing settings
>
> cache-min-ttl: 600
> cache-max-ttl: 43200
> cache-max-negative-ttl: 100
>
> when it works as expected it should lead in not so often expire heavily
> used crap domains without take too long for realize new listings and at
> least makes the problem nit so big as now
sadly it don't work as expected
https://bugzilla.redhat.com/show_bug.cgi?id=1360222
Re: URIBL randomly not triggered for the same message
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.07.2016 um 17:35 schrieb John Hardin:
> On Wed, 6 Jul 2016, Paul Stead wrote:
>
>> On 06/07/16 16:16, John Hardin wrote:
>>> Does that cache-min-ttl also affect NXDOMAIN? Is it possible to
>>> configure different TTL for NXDOMAIN (relatively low) and positive
>>> results (relatively high)?
>>
>> For this cache-max-negative-ttl exists :)
>
> :) It's obvious I don't use unbound...
>
> Reindl, does that approach help?
sounds good and at leat i don't get any error by using
unbound-1.5.8-2.fc23.x86_64 and the follwoing settings
cache-min-ttl: 600
cache-max-ttl: 43200
cache-max-negative-ttl: 100
when it works as expected it should lead in not so often expire heavily
used crap domains without take too long for realize new listings and at
least makes the problem nit so big as now
thanks god then normal DNSBL/DNSWL are not affected becaus ethey are
used also in prostscreen for weighting and so at the moment SA is using
them the cache is always hot
Re: URIBL randomly not triggered for the same message
Posted by John Hardin <jh...@impsec.org>.
On Wed, 6 Jul 2016, Paul Stead wrote:
> On 06/07/16 16:16, John Hardin wrote:
>> Does that cache-min-ttl also affect NXDOMAIN? Is it possible to
>> configure different TTL for NXDOMAIN (relatively low) and positive
>> results (relatively high)?
>
> For this cache-max-negative-ttl exists :)
:) It's obvious I don't use unbound...
Reindl, does that approach help?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
So Microsoft's invented the ASCII equivalent to ugly ink spots that
appear on your letter when your pen is malfunctioning.
-- Greg Andrews, about Microsoft's way to encode apostrophes
-----------------------------------------------------------------------
Tomorrow: Robert Heinlein's 109th birthday
Re: URIBL randomly not triggered for the same message
Posted by Paul Stead <pa...@zeninternet.co.uk>.
On 06/07/16 16:16, John Hardin wrote:
> Does that cache-min-ttl also affect NXDOMAIN? Is it possible to
> configure different TTL for NXDOMAIN (relatively low) and positive
> results (relatively high)?
For this cache-max-negative-ttl exists :)
Paul
--
Paul Stead
Systems Engineer
Zen Internet
Re: URIBL randomly not triggered for the same message
Posted by John Hardin <jh...@impsec.org>.
On Wed, 6 Jul 2016, Reindl Harald wrote:
>
>
> Am 06.07.2016 um 14:36 schrieb RW:
>> On Tue, 5 Jul 2016 14:01:17 +0200
>> Reindl Harald wrote:
>>
>> > since there is a local unbound-cache with
>> >
>> > cache-min-ttl: 300
>
> thanks for the hint, but look at
> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7335#c8
>
> reduce the value would make the problem even worser because what i observe is
> that after TTL is reached and unbound needs to query again the at least first
> question leads to a negativeresult in spamassassin while the next cache hit
> correctly has URIBL_BLACK again
Does that cache-min-ttl also affect NXDOMAIN? Is it possible to configure
different TTL for NXDOMAIN (relatively low) and positive results
(relatively high)?
If not, you might want to file a bug with unbound to ask them to make that
possible.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
So Microsoft's invented the ASCII equivalent to ugly ink spots that
appear on your letter when your pen is malfunctioning.
-- Greg Andrews, about Microsoft's way to encode apostrophes
-----------------------------------------------------------------------
Tomorrow: Robert Heinlein's 109th birthday
Re: URIBL randomly not triggered for the same message
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.07.2016 um 14:36 schrieb RW:
> On Tue, 5 Jul 2016 14:01:17 +0200
> Reindl Harald wrote:
>
>> since there is a local unbound-cache with
>>
>> cache-min-ttl: 300
thanks for the hint, but look at
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7335#c8
reduce the value would make the problem even worser because what i
observe is that after TTL is reached and unbound needs to query again
the at least first question leads to a negativeresult in spamassassin
while the next cache hit correctly has URIBL_BLACK again
so at the moment there is a tradeoff between get new domains fast enough
and don't miss already known hits *and* that also affects SPF and so
whitelist_auth in a bad way
> You might want to review that. From http://uribl.com
>
> July 8, 2015: Reduction in list time latency
>
> The spam trend of late has been to use short lived, high-volume
> campaigns in order to capitalize on the reactive nature of blacklist
> services. In the past, it could take up to 4 minutes for us to
> identify, list, rebuild, and syncronize the update. Recent campaigns
> we have investigated have sent 80-90% of their payload within 3
> minutes.
>
> Because of this, we have made a handful of enhancements to improve
> our identification speed and reduce the list time latency. As a
> result, we have reduced identification times by up to 100 seconds for
> new spam campaigns, by improving the speed at which we deliver live
> query data into our system. All users should see immediate results
> from these changes.
Re: URIBL randomly not triggered for the same message
Posted by RW <rw...@googlemail.com>.
On Tue, 5 Jul 2016 14:01:17 +0200
Reindl Harald wrote:
> since there is a local unbound-cache with
>
> cache-min-ttl: 300
You might want to review that. From http://uribl.com
July 8, 2015: Reduction in list time latency
The spam trend of late has been to use short lived, high-volume
campaigns in order to capitalize on the reactive nature of blacklist
services. In the past, it could take up to 4 minutes for us to
identify, list, rebuild, and syncronize the update. Recent campaigns
we have investigated have sent 80-90% of their payload within 3
minutes.
Because of this, we have made a handful of enhancements to improve
our identification speed and reduce the list time latency. As a
result, we have reduced identification times by up to 100 seconds for
new spam campaigns, by improving the speed at which we deliver live
query data into our system. All users should see immediate results
from these changes.
Re: URIBL randomly not triggered (and SPF too)
Posted by Reindl Harald <h....@thelounge.net>.
see also https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7335
BTW: the bugtracker has also a major bug - click on "My Bugs" leads to
the URL below listing a ton of bugreports back to the year 2011 and
pretends they are reported by me
https://bz.apache.org/SpamAssassin/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailreporter1=1&emailtype1=exact&%20%20%20%20%20%20%20%20%20email1=h.reindl%40thelounge.net&field0-0-0=bug_status&type0-0-0=notequals&value0-0-0=UNCONFIRMED&field0-0-1=reporter&type0-0-1=equals&value0-0-1=h.reindl%40thelounge.net
Am 05.07.2016 um 14:10 schrieb Reindl Harald:
> Am 05.07.2016 um 14:01 schrieb Reindl Harald:
>> i have here a message with URIBL_ABUSE_SURBL Contains an URL listed in
>> the ABUSE SURBL blocklist
>>
>> 50% of all tries against spamd it does NOT hit while the scantime for
>> the whole message is arounnd 3 seconds - since there is a local
>> unbound-cache with
>>
>> cache-min-ttl: 300
>> cache-max-ttl: 10800
>>
>> it's impossible that there are happening dns timeouts and i can observe
>> the same behavior randomly with URIBL_LOCAL where the unbound dns cache
>> on 127.0.0.1:53 talks to rblsdnsd on 127.0.0.1:1053
>>
>> that smells why ever very unrelieable and frankly i observed similar
>> with SPF_PASS / SHORTCIRCUIT where people within 5 seconds get the same
>> message and one get USER_IN_SPF_WHITELIST while the other goes through
>> all tests
>
> that below too MUST NOT happen because one triggers
> USER_IN_SPF_WHITELIST and the other don't have any SPF test and given
> that there is a python-policyd-spf waiting 20 seconds for the response
> in 'smtpd_recipient_restrictions' long before the contentfilters the
> dns-results are cached
>
> Jul 4 11:34:51 mail-gw postfix/smtpd[13648]: 3rjhgb71LVzB47:
> client=o3.email.wetransfer.com[192.254.123.42]
> Jul 4 11:34:52 mail-gw spamd[12535]: spamd: processing message
> <57...@delayedjobs-17aj6hbldm9spghikobe88v7k.wetransfer.com.mail>
> for sa-milt:189
> Jul 4 11:34:56 mail-gw spamd[12535]: spamd: result: . -4 -
> BAYES_00,CUST_DNSWL_2_SENDERSC_L,CUST_DNSWL_3_JEF_L,CUST_DNSWL_5_ORG_N,CUST_DNSWL_8_TL_N,DKIM_SIGNED,DKIM_VALID,HTML_MESSAGE,RCVD_IN_MSPIKE_H2,RP_MATCHES_RCVD
> scantime=4.2,size=18438,user=sa-milt,uid=189,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=/run/spamassassin/spamassassin.sock,mid=<57...@delayedjobs-17aj6hbldm9spghikobe88v7k.wetransfer.com.mail>,bayes=0.000000,autolearn=disabled,shortcircuit=no
>
>
> Jul 4 11:57:01 mail-gw postfix/smtpd[14837]: 3rjj993Bk8zB7P:
> client=o3.email.wetransfer.com[192.254.123.42]
> Jul 4 11:57:02 mail-gw spamd[14302]: spamd: processing message
> <57...@delayedjobs-16gux7nsdp9xgp69boio5hcsg.wetransfer.com.mail>
> for sa-milt:189
> Jul 4 11:57:02 mail-gw spamd[14302]: spamd: result: . -100 -
> CUST_DNSWL_2_SENDERSC_L,CUST_DNSWL_3_JEF_L,CUST_DNSWL_5_ORG_N,CUST_DNSWL_8_TL_N,CUST_SHORTCIRCUIT,RCVD_IN_MSPIKE_H2,SHORTCIRCUIT,USER_IN_SPF_WHITELIST
> scantime=0.1,size=15685,user=sa-milt,uid=189,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=/run/spamassassin/spamassassin.sock,mid=<57...@delayedjobs-16gux7nsdp9xgp69boio5hcsg.wetransfer.com.mail>,autolearn=disabled,shortcircuit=spam
Re: URIBL randomly not triggered (and SPF too)
Posted by Reindl Harald <h....@thelounge.net>.
Am 05.07.2016 um 14:01 schrieb Reindl Harald:
> i have here a message with URIBL_ABUSE_SURBL Contains an URL listed in
> the ABUSE SURBL blocklist
>
> 50% of all tries against spamd it does NOT hit while the scantime for
> the whole message is arounnd 3 seconds - since there is a local
> unbound-cache with
>
> cache-min-ttl: 300
> cache-max-ttl: 10800
>
> it's impossible that there are happening dns timeouts and i can observe
> the same behavior randomly with URIBL_LOCAL where the unbound dns cache
> on 127.0.0.1:53 talks to rblsdnsd on 127.0.0.1:1053
>
> that smells why ever very unrelieable and frankly i observed similar
> with SPF_PASS / SHORTCIRCUIT where people within 5 seconds get the same
> message and one get USER_IN_SPF_WHITELIST while the other goes through
> all tests
that below too MUST NOT happen because one triggers
USER_IN_SPF_WHITELIST and the other don't have any SPF test and given
that there is a python-policyd-spf waiting 20 seconds for the response
in 'smtpd_recipient_restrictions' long before the contentfilters the
dns-results are cached
Jul 4 11:34:51 mail-gw postfix/smtpd[13648]: 3rjhgb71LVzB47:
client=o3.email.wetransfer.com[192.254.123.42]
Jul 4 11:34:52 mail-gw spamd[12535]: spamd: processing message
<57...@delayedjobs-17aj6hbldm9spghikobe88v7k.wetransfer.com.mail>
for sa-milt:189
Jul 4 11:34:56 mail-gw spamd[12535]: spamd: result: . -4 -
BAYES_00,CUST_DNSWL_2_SENDERSC_L,CUST_DNSWL_3_JEF_L,CUST_DNSWL_5_ORG_N,CUST_DNSWL_8_TL_N,DKIM_SIGNED,DKIM_VALID,HTML_MESSAGE,RCVD_IN_MSPIKE_H2,RP_MATCHES_RCVD
scantime=4.2,size=18438,user=sa-milt,uid=189,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=/run/spamassassin/spamassassin.sock,mid=<57...@delayedjobs-17aj6hbldm9spghikobe88v7k.wetransfer.com.mail>,bayes=0.000000,autolearn=disabled,shortcircuit=no
Jul 4 11:57:01 mail-gw postfix/smtpd[14837]: 3rjj993Bk8zB7P:
client=o3.email.wetransfer.com[192.254.123.42]
Jul 4 11:57:02 mail-gw spamd[14302]: spamd: processing message
<57...@delayedjobs-16gux7nsdp9xgp69boio5hcsg.wetransfer.com.mail>
for sa-milt:189
Jul 4 11:57:02 mail-gw spamd[14302]: spamd: result: . -100 -
CUST_DNSWL_2_SENDERSC_L,CUST_DNSWL_3_JEF_L,CUST_DNSWL_5_ORG_N,CUST_DNSWL_8_TL_N,CUST_SHORTCIRCUIT,RCVD_IN_MSPIKE_H2,SHORTCIRCUIT,USER_IN_SPF_WHITELIST
scantime=0.1,size=15685,user=sa-milt,uid=189,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=/run/spamassassin/spamassassin.sock,mid=<57...@delayedjobs-16gux7nsdp9xgp69boio5hcsg.wetransfer.com.mail>,autolearn=disabled,shortcircuit=spam