You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Danny Angus (JIRA)" <se...@james.apache.org> on 2007/01/11 12:13:27 UTC
[jira] Created: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
InSpammerBlacklist latency seriously affects throughput
-------------------------------------------------------
Key: JAMES-758
URL: https://issues.apache.org/jira/browse/JAMES-758
Project: James
Issue Type: Improvement
Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
Affects Versions: 2.3.0, 2.3.0rc5, 2.3.0rc4, 2.3.0rc3, 2.3.0rc2, 2.3.0rc1, 2.3.0b3, 2.3.0b2, 2.3.0b1, 2.3.0a3, 2.3.0a2, 2.3.0a1, 2.2.0, 2.3.1-dev, Next Minor, Next Major, Trunk
Environment: nothing fancy, but network latency may be a contributing factor
Reporter: Danny Angus
the following are sample mean timings from the InSpammerBlacklist matcher:
all requests were for 127.0.0.1
hostname - lookup count - mean timing - this lookup time
1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
As you can see they all take signifcant time, and ordb.org is painful.
Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
Not good.
Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Assigned: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
Posted by "Danny Angus (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/JAMES-758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Danny Angus reassigned JAMES-758:
---------------------------------
Assignee: Danny Angus
> InSpammerBlacklist latency seriously affects throughput
> -------------------------------------------------------
>
> Key: JAMES-758
> URL: https://issues.apache.org/jira/browse/JAMES-758
> Project: James
> Issue Type: Improvement
> Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
> Affects Versions: 2.2.0, 2.3.0a1, 2.3.0a2, 2.3.0a3, 2.3.0b1, 2.3.0b2, 2.3.0b3, 2.3.0rc1, 2.3.0rc2, 2.3.0rc3, 2.3.0rc4, 2.3.0rc5, 2.3.0, 2.3.1-dev, Next Minor, Next Major, Trunk
> Environment: nothing fancy, but network latency may be a contributing factor
> Reporter: Danny Angus
> Assigned To: Danny Angus
>
> the following are sample mean timings from the InSpammerBlacklist matcher:
> all requests were for 127.0.0.1
> hostname - lookup count - mean timing - this lookup time
> 1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
> 1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
> 1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
> As you can see they all take signifcant time, and ordb.org is painful.
> Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
> We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
> In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
> Not good.
> Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
Posted by "Amichai Rothman (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/JAMES-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466347 ]
Amichai Rothman commented on JAMES-758:
---------------------------------------
Two more thoughts:
1. if ordb.org, when still configured, is causing critical delays, JAMES users should be notified of this update somehow and advised to remove it.
2. it's a good opportunity to update the default/recommended blacklist services. I don't know much about them, though I've seen recommendations for the servers below. If anyone knows more about these and others, pls make sure the good ones are added...
sbl-xbl.spamhaus.org,
list.dsbl.org,
bl.spamcop.net
> InSpammerBlacklist latency seriously affects throughput
> -------------------------------------------------------
>
> Key: JAMES-758
> URL: https://issues.apache.org/jira/browse/JAMES-758
> Project: James
> Issue Type: Improvement
> Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
> Affects Versions: 2.2.0, 2.3.0, 2.3.1-dev, Next Minor, Next Major, Trunk
> Environment: nothing fancy, but network latency may be a contributing factor
> Reporter: Danny Angus
> Assigned To: Danny Angus
>
> the following are sample mean timings from the InSpammerBlacklist matcher:
> all requests were for 127.0.0.1
> hostname - lookup count - mean timing - this lookup time
> 1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
> 1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
> 1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
> As you can see they all take signifcant time, and ordb.org is painful.
> Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
> We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
> In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
> Not good.
> Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
Posted by "Stefano Bagnara (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/JAMES-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465701 ]
Stefano Bagnara commented on JAMES-758:
---------------------------------------
I didn't see this issue before: today I had to remove "relays.ordb.org." from my config.xml in order to let my mail to pass through. I had thousands of mails waiting for dns resolutions.
I think this is getting critical and we should probably change our default config.xml to remove by default that mailet and to add a comment about current timings of that hosts.
> InSpammerBlacklist latency seriously affects throughput
> -------------------------------------------------------
>
> Key: JAMES-758
> URL: https://issues.apache.org/jira/browse/JAMES-758
> Project: James
> Issue Type: Improvement
> Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
> Affects Versions: 2.2.0, 2.3.0, 2.3.1-dev, Next Minor, Next Major, Trunk
> Environment: nothing fancy, but network latency may be a contributing factor
> Reporter: Danny Angus
> Assigned To: Danny Angus
>
> the following are sample mean timings from the InSpammerBlacklist matcher:
> all requests were for 127.0.0.1
> hostname - lookup count - mean timing - this lookup time
> 1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
> 1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
> 1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
> As you can see they all take signifcant time, and ordb.org is painful.
> Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
> We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
> In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
> Not good.
> Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Resolved: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
Posted by "Norman Maurer (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/JAMES-758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Norman Maurer resolved JAMES-758.
---------------------------------
Resolution: Fixed
Fix Version/s: 3.0-M1
was removed from config.xml
> InSpammerBlacklist latency seriously affects throughput
> -------------------------------------------------------
>
> Key: JAMES-758
> URL: https://issues.apache.org/jira/browse/JAMES-758
> Project: JAMES Server
> Issue Type: Improvement
> Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
> Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2, 3.0, Trunk
> Environment: nothing fancy, but network latency may be a contributing factor
> Reporter: Danny Angus
> Assignee: Danny Angus
> Fix For: 3.0-M1
>
>
> the following are sample mean timings from the InSpammerBlacklist matcher:
> all requests were for 127.0.0.1
> hostname - lookup count - mean timing - this lookup time
> 1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
> 1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
> 1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
> As you can see they all take signifcant time, and ordb.org is painful.
> Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
> We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
> In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
> Not good.
> Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[jira] Commented: (JAMES-758) InSpammerBlacklist latency seriously
affects throughput
Posted by "Amichai Rothman (JIRA)" <se...@james.apache.org>.
[ https://issues.apache.org/jira/browse/JAMES-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466343 ]
Amichai Rothman commented on JAMES-758:
---------------------------------------
>From a quick search, it appears that ordb.org has been shut down permanently. this might be part of the problem (there still may be latency issues with existing blacklist servers - still needs to be checked). pls remove ordb.org from both the default configuration, and all the docs, and anywhere else it may appear...
:-(
> InSpammerBlacklist latency seriously affects throughput
> -------------------------------------------------------
>
> Key: JAMES-758
> URL: https://issues.apache.org/jira/browse/JAMES-758
> Project: James
> Issue Type: Improvement
> Components: DNSServer, Matchers/Mailets (bundled), SpoolManager & Processors
> Affects Versions: 2.2.0, 2.3.0, 2.3.1-dev, Next Minor, Next Major, Trunk
> Environment: nothing fancy, but network latency may be a contributing factor
> Reporter: Danny Angus
> Assigned To: Danny Angus
>
> the following are sample mean timings from the InSpammerBlacklist matcher:
> all requests were for 127.0.0.1
> hostname - lookup count - mean timing - this lookup time
> 1.0.0.127.query.bondedsender.org. - 10.0 - 71.9 - 16.0
> 1.0.0.127.dnsbl.njabl.org. - 10.0 - 265.6 - 266.0
> 1.0.0.127.relays.ordb.org. - 10.0 - 20095.6 - 20172.0
> As you can see they all take signifcant time, and ordb.org is painful.
> Of course "success" of the matcher is !success of the lookup, which means that while we will cache the hits they are only the failures, and the good mail will have to perform a full ns lookup everytime.
> We should think about caching the successes locally, or arranging these mailets in a separate processor and having independantly threaded processors.
> In this case (ten mails) each thread paused for 20 seconds waiting for ordb and then continued, making the whole thing pause for 20. But because I was running 10 spool threads when I increased this to 20 mails the threads paused twice, meaning that the 21st message took 40+ seconds to complete its journey through the spoolmanager.
> Not good.
> Workaround, don't use InSpammerBlacklist or set it up in a processor so that it is called by ToProcessor only on classes of mail for which the expense is justified (e.g. not on any outbound or on trusted IP's, senders or domains)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org