You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2007/09/19 23:02:12 UTC
Re: Non-DNS async support
if I recall correctly, most of the code works fine without using
DNS; it's just that most of the code currently *using* it works
with DNS.
--j.
Giampaolo Tomassoni writes:
> Dears,
>
> my belief was that the Async support in SA was somehow of general-purpose
> type.
>
> I see instead that most code in AsyncLoop.pm do rely on code in
> DnsResolver.pm.
>
> Willing to handle non-DNS connection asynchronously in SA, how am I expected
> to obtain it?
>
> Thanks,
>
> Giampaolo
R: Non-DNS async support
Posted by Giampaolo Tomassoni <g....@libero.it>.
> -----Messaggio originale-----
> Da: Mark Martinec [mailto:Mark.Martinec+sa@ijs.si]
>
> Giampaolo,
>
> > Well, I have 3.2.1 and the excerpt from AsyncLoop.pm was from there.
> > But anyway, how is supposed to be set the timeout value of a non-DNS
> query?
>
> The current code in trunk is able to specify and honour individual
> timeouts for each async request - and it defaults to rbl_timeout
> if not specified otherwise. See sub AsyncLoop::start_lookup()
> and $ent->{timeout} attribute in an object passed to it.
>
> > Maybe my code stops due to a timeout: messages are non that clear...
>
> See http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5589
>
> The current code in trunk deals with timeouts more accurately.
> The patch in Bug 5589 can be applied to 3.2.3, if one wants
> to avoid running the bleeding edge trunk code.
>
> > It may even be a timeout, then. It seems to me there is no way to set
> a
> > lookup timeout in start_lookup() in AsyncLoop.pm. Right?
>
> True in 3.2.3, not true in (3.3.0)SVN.
>
> > By the way, it may be that the Async code is undergoing many changes.
> > Is there any SA version in which it could be regarded as stable?
>
> For doing new development it is best to start with the current code in
> trunk, otherwise one could be solving problems which are already
> solved.
> Of course running the leading edge code bears its risks and offers no
> guarantees (but there are no real guarantees for 3.2.3 either,
> right?!),
> so one should be prepared to peek into code and solve some glitch if
> need arises - and subscribing to a 'dev' mailing list is advised.
>
> Nevertheless, some people do run the trunk code in their test or even
> in production environment. Generally the trunk code is supposed to
> always be runnable on a mainstream environment - e.g. Perl 5.8.8 on
> Unix,
> with recent versions of external Perl modules. If running older Perl
> or being on Windows, chances are much higher that some feature is
> not yet thouroughly tested. Mishaps do happen on occasion, but are
> usually sorted in a day or two, and reverting to a revision before
> a breakage is always a quick-fix workaround. The decision mostly
> depend on your willingness to get hands dirty on occasion, benefits
> are that there is a quickest response to problems, old and new.
> In my experience the current trunk is well behaved and quite stable
> as it stands at the moment, and is still compatible with 3.2.3,
> so one can revert to 3.2.3 in an emergency.
Mark,
thank you for your precious hints: I found SA 3.2.3 to fix the matter.
As you told me, however, I can't still specify a timeout in start_lookup().
Anyway, this isn't very important, because it seems to me that 3.2.3 raises
the default to 6 seconds, which is pretty fine.
I'll stick to 3.2.3 since I'm going to put this plugin into a production
server and of course I would prefer not to tie it to "trunk" code.
If I correctly understand you reply, the Async API didn't change too much in
the trunk (apart for the timeout enhancement), thereby I'm going to expect
this plugin to work against trunk too. If this will not be the case, I'll
adjust to the future needs when they'll get out...
Thank you again,
Giampaolo
>
> Mark
Re: Non-DNS async support
Posted by Mark Martinec <Ma...@ijs.si>.
Giampaolo,
> Well, I have 3.2.1 and the excerpt from AsyncLoop.pm was from there.
> But anyway, how is supposed to be set the timeout value of a non-DNS query?
The current code in trunk is able to specify and honour individual
timeouts for each async request - and it defaults to rbl_timeout
if not specified otherwise. See sub AsyncLoop::start_lookup()
and $ent->{timeout} attribute in an object passed to it.
> Maybe my code stops due to a timeout: messages are non that clear...
See http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5589
The current code in trunk deals with timeouts more accurately.
The patch in Bug 5589 can be applied to 3.2.3, if one wants
to avoid running the bleeding edge trunk code.
> It may even be a timeout, then. It seems to me there is no way to set a
> lookup timeout in start_lookup() in AsyncLoop.pm. Right?
True in 3.2.3, not true in (3.3.0)SVN.
> By the way, it may be that the Async code is undergoing many changes.
> Is there any SA version in which it could be regarded as stable?
For doing new development it is best to start with the current code in
trunk, otherwise one could be solving problems which are already solved.
Of course running the leading edge code bears its risks and offers no
guarantees (but there are no real guarantees for 3.2.3 either, right?!),
so one should be prepared to peek into code and solve some glitch if
need arises - and subscribing to a 'dev' mailing list is advised.
Nevertheless, some people do run the trunk code in their test or even
in production environment. Generally the trunk code is supposed to
always be runnable on a mainstream environment - e.g. Perl 5.8.8 on Unix,
with recent versions of external Perl modules. If running older Perl
or being on Windows, chances are much higher that some feature is
not yet thouroughly tested. Mishaps do happen on occasion, but are
usually sorted in a day or two, and reverting to a revision before
a breakage is always a quick-fix workaround. The decision mostly
depend on your willingness to get hands dirty on occasion, benefits
are that there is a quickest response to problems, old and new.
In my experience the current trunk is well behaved and quite stable
as it stands at the moment, and is still compatible with 3.2.3,
so one can revert to 3.2.3 in an emergency.
Mark
R: R: Non-DNS async support
Posted by Giampaolo Tomassoni <g....@libero.it>.
> -----Messaggio originale-----
> Da: Mark Martinec [mailto:Mark.Martinec+sa@ijs.si]
>
> Giampaolo,
>
>
> Not really (at least not in the current trunk I'm looking at).
> As long as there are any (non timed-out) requests in %$pending,
> it does not terminate, even if poll returns empty-handed.
Well, I have 3.2.1 and the excerpt from AsyncLoop.pm was from there.
But anyway, how is supposed to be set the timeout value of a non-DNS query?
Maybe my code stops due to a timeout: messages are non that clear...
dbg: uri_whois: Resolved 61994/whois.verisign-grs.com/A/IN to 199.7.51.74
dbg: uri_whois: Starting whois query for yahoo.com to
whois.verisign-grs.com (199.7.51.74)
dbg: async: queries completed: 1 started: 1
dbg: async: queries active: DNSBL-A=2 URIWHOIS-A=1 at Wed Sep 19 18:02:13
2007
dbg: async: select found no socks ready
dbg: uri_whois: Connected whois query for yahoo.com to
whois.verisign-grs.com
dbg: async: queries completed: 0 started: 0
dbg: async: queries active: DNSBL-A=2 URIWHOIS-A=1 URIWHOIS-WHOIS=1 at
Wed Sep 19 18:02:14 2007
dbg: dns: success for 23 of 27 queries
| dbg: dns: timeout for after 1 seconds
| dbg: dns: timeout for whois after 1 seconds
| dbg: dns: timeout for after 1 seconds
| dbg: dns: timeout for whois,whois-lastexternal after 1 seconds
It may even be a timeout, then. It seems to me there is no way to set a
lookup timeout in start_lookup() in AsyncLoop.pm. Right?
By the way, it may be that the Async code is undergoing many changes. Is
there any SA version in which it could be regarded as stable?
Thanks,
Giampaolo
> And 'a request' is a rather general term. Currently it covers
> DNS requests, but in principle could be other kind of 'requests'.
> There are routines for submitting and terminating requests.
>
> I'd be interested too if more general usage emerges, perhaps
> along with a somewhat cleaned/generalized API. My wish is
> to provide a mechanism for Plugins (such as DKIM, SPF, Razor2)
> to use the same async services and avoid doing their network
> lookups synchronously on their own. Suggestions welcome.
> If only Perl could have a way of expressing co-routines,
> then one could build a simple task scheduler, with task queues
> on blocked resources, etc.
>
> > Does anybody knows which is the API to the async stuff? Maybe
> > I didn't get it at all and I'm attempting to use it the wrong way...
>
> Not much docs about API, but see comments on subroutines in
> AsyncLoop.pm, and their typical usage by Dns.pm.
>
> Mark
Re: R: Non-DNS async support
Posted by Mark Martinec <Ma...@ijs.si>.
Giampaolo,
> Almost. One of the biggest problems I see is that one can't register a
> socket with AsyncLoop.pm. Thereby, when DNS lookups are finished, the whole
> loop stops regardless of outstanding non-DNS-based lookups.
>
> See sub complete_lookups in AsyncLoop.pm:
> my $nfound = $self->{main}->{resolver}->poll_responses($timeout);
> $nfound ||= 'no';
> dbg ("async: select found $nfound socks ready");
> foreach my $key (keys %{$pending}) {
> my $ent = $pending->{$key};
>
> poll_responsens is in DnsResolver.pm and yield 0 when DNS queries are
> finished. This causes the complete_lookups call to terminate all the
> outstanding lookups, even the non-DNS-based ones.
Not really (at least not in the current trunk I'm looking at).
As long as there are any (non timed-out) requests in %$pending,
it does not terminate, even if poll returns empty-handed.
And 'a request' is a rather general term. Currently it covers
DNS requests, but in principle could be other kind of 'requests'.
There are routines for submitting and terminating requests.
I'd be interested too if more general usage emerges, perhaps
along with a somewhat cleaned/generalized API. My wish is
to provide a mechanism for Plugins (such as DKIM, SPF, Razor2)
to use the same async services and avoid doing their network
lookups synchronously on their own. Suggestions welcome.
If only Perl could have a way of expressing co-routines,
then one could build a simple task scheduler, with task queues
on blocked resources, etc.
> Does anybody knows which is the API to the async stuff? Maybe
> I didn't get it at all and I'm attempting to use it the wrong way...
Not much docs about API, but see comments on subroutines in
AsyncLoop.pm, and their typical usage by Dns.pm.
Mark
R: Non-DNS async support
Posted by Giampaolo Tomassoni <g....@libero.it>.
> if I recall correctly, most of the code works fine without using
> DNS; it's just that most of the code currently *using* it works
> with DNS.
Almost. One of the biggest problems I see is that one can't register a
socket with AsyncLoop.pm. Thereby, when DNS lookups are finished, the whole
loop stops regardless of outstanding non-DNS-based lookups.
See sub complete_lookups in AsyncLoop.pm:
eval {
my $nfound = $self->{main}->{resolver}->poll_responses($timeout);
$nfound ||= 'no';
dbg ("async: select found $nfound socks ready");
foreach my $key (keys %{$pending}) {
my $ent = $pending->{$key};
poll_responsens is in DnsResolver.pm and yield 0 when DNS queries are
finished. This causes the complete_lookups call to terminate all the
outstanding lookups, even the non-DNS-based ones.
Does anybody knows which is the API to the async stuff? Maybe I didn't get
it at all and I'm attempting to use it the wrong way...
Giampaolo
>
> --j.
>
> Giampaolo Tomassoni writes:
> > Dears,
> >
> > my belief was that the Async support in SA was somehow of general-
> purpose
> > type.
> >
> > I see instead that most code in AsyncLoop.pm do rely on code in
> > DnsResolver.pm.
> >
> > Willing to handle non-DNS connection asynchronously in SA, how am I
> expected
> > to obtain it?
> >
> > Thanks,
> >
> > Giampaolo