You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by JT DeLys <jt...@gmail.com> on 2007/06/25 20:56:50 UTC
SA fails to search specified DATADIR for Distribution files
Exploring some problems I've been having, I have uploaded the latest
SpamAssassin SVN source code.
I'm specifying 'custom' locations for SpamAssassin local, distribution &
update files.
Despite being correctly 'told' where to pick up the files, SA seems to
ignore the DATADIR spec (where the Distribution files are), only checking in
the LOCALSTATEDIR (where the UPdates are), and traversing no further.
Can someone help find the problem?
I've pasted some of the details below.
Thanks,
JTDeLys
svn info
Path: .
URL: http://svn.apache.org/repos/asf/spamassassin/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 550563
Node Kind: directory
Schedule: normal
Last Changed Author: jm
Last Changed Rev: 550555
Last Changed Date: 2007-06-25 10:20:48 -0700 (Mon, 25 Jun 2007)
Reading PACKAGING, the wiki, and nformation from previous list posts,
DATADIR (DEFRULESDIR):
SpamAssassin's real logic lies in its shipped rule definitions and the
corresponding scores. The files with these settings have to be saved
somewhere, normally below PREFIX/share/spamassassin. The full path to
that directory can be changed with this variable (DEFRULESDIR is a
synonym).
CONFDIR (LOCALRULESDIR):
SpamAssassin looks for its config files in SYSCONFDIR/mail/spamassassin.
LOCALSTATEDIR:
"sa-update" will download rule updates into LOCALSTATEDIR/spamassassin.
I've chosen to configure & build using,
perl Makefile.PL \
PREFIX=/usr/local/spamassassin \
DATADIR=/usr/local/etc/spamassassin/Distribution \
CONFDIR=/usr/local/etc/spamassassin/Local \
LOCALSTATEDIR=/usr/local/etc/spamassassin/Updates
cd spamc
perl version.h.pl
After install, sa-update (of both Distribution & SARE rules), and
sa-compile, I have
cd /usr/local/etc/spamassassin
ls Distribution/
10_default_prefs.cf 20_vbounce.cf 30_text_it.cf
20_advance_fee.cf 23_bayes.cf 30_text_nl.cf
20_body_tests.cf 25_accessdb.cf 30_text_pl.cf
20_compensate.cf 25_antivirus.cf 30_text_pt_br.cf
20_dnsbl_tests.cf 25_asn.cf 50_scores.cf
20_drugs.cf 25_dcc.cf 60_awl.cf
20_dynrdns.cf 25_dkim.cf 60_shortcircuit.cf
20_fake_helo_tests.cf 25_domainkeys.cf 60_whitelist.cf
20_head_tests.cf 25_hashcash.cf 60_whitelist_dk.cf
20_html_tests.cf 25_pyzor.cf 60_whitelist_dkim.cf
20_imageinfo.cf 25_razor2.cf 60_whitelist_spf.cf
20_meta_tests.cf 25_replace.cf 60_whitelist_subject.cf
20_net_tests.cf 25_spf.cf languages
20_phrases.cf 25_textcat.cf sa-update-pubkey.txt
20_porn.cf 25_uribl.cf user_prefs.template
20_ratware.cf 30_text_de.cf
20_uri_tests.cf 30_text_fr.cf
ls -rd Updates/*
Updates/compiled Updates/3.003000
Next, checking in the resulting "man spamassassin",
...
CONFIGURATION FILES
...
Default configuration data is loaded from the first existing
directory
in:
/usr/local/etc/spamassassin/Updates/3.003000
/usr/local/etc/spamassassin/Distribution
/usr/local/spamassassin/share/spamassassin
/usr/local/share/spamassassin
/usr/share/spamassassin
Site-specific configuration data is used to override any values which
had already been set. This is loaded from the first existing
directory
in:
/usr/local/etc/spamassassin/Local
/usr/local/spamassassin/etc/mail/spamassassin
/usr/local/spamassassin/etc/spamassassin
/usr/local/etc/spamassassin
/usr/pkg/etc/spamassassin
/usr/etc/spamassassin
/etc/mail/spamassassin
/etc/spamassassin
...
The problem is that
spamassassin --lint --nocreate-prefs --debug > /tmp/sa_debug_output.txt
reports that,
[24122] dbg: config: using "/usr/local/etc/spamassassin/Local" for site
rules pre files
[24122] dbg: config: read file /usr/local/etc/spamassassin/Local/init.pre
[24122] dbg: config: using "/usr/local/etc/spamassassin/Updates/3.003000"
for sys rules pre files
[24122] dbg: config: using "/usr/local/etc/spamassassin/Updates/3.003000"
for default rules dir
and, apparently, never picks up the original .cf's from,
/usr/local/etc/spamassassin/Distribution
so, e.g.,
cd /usr/local/etc/spamassassin
find . | grep -i 50_
./Distribution/50_scores.cf
never gets found/read -- DESPITE having assigned
DATADIR=/usr/local/etc/spamassassin/Distribution
It seems to me that, according to,
" Default configuration data is loaded from the first existing
directory
in:
"
the Default config data is NOT picked up from the first INSTANCE of a given
.cf in the search hierarchy, just from the first DIR listed there.
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by Theo Van Dinter <fe...@apache.org>.
On Wed, Jun 27, 2007 at 04:30:12PM +0100, Nix wrote:
> But I'm glad to hear that it's meant to work and is working :) I guess
> I hit a glitch in the updates-updating process.
IIRC, there were some issues back before 3.2.0 was released, where the
3.2 branch was different than trunk but wasn't considered "stable" yet,
and people were using it, and updates may have been problematic depending
on the mix of update and code. But now that it is the stable branch,
it should always work.
:)
--
Randomly Selected Tagline:
"The distributions may be developed that way, but to get into the mainline
kernel you have to go through the pope." - David Parsons
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
> It depends on the version that the 3.2-branch announces itself as, I
> think, since that determines the DNS name that's checked within the
> SA channel.
Good point.
Then, just FYI, here (for the moment),
> spamassassin --version
SpamAssassin version 3.2.2-r547235
running on Perl version 5.8.8
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by Nix <ni...@esperi.org.uk>.
On 27 Jun 2007, JT DeLys said:
>> The same thing is true if you install from the 3.2-branch, or at least
>> it was in the 3.2.0 days
>
> I'm currently running a build from svn 3.2-branch, and it /seems/ to
> be working correctly. There are, at least, files being pulled into my
> LOCALSTATEDIR for both SARE & SA-Distrib rules.
It depends on the version that the 3.2-branch announces itself as, I
think, since that determines the DNS name that's checked within the
SA channel.
But I'm glad to hear that it's meant to work and is working :) I guess
I hit a glitch in the updates-updating process.
--
`... in the sense that dragons logically follow evolution so they would
be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
furiously
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
> The same thing is true if you install from the 3.2-branch, or at least
> it was in the 3.2.0 days
I'm currently running a build from svn 3.2-branch, and it /seems/ to
be working correctly. There are, at least, files being pulled into my
LOCALSTATEDIR for both SARE & SA-Distrib rules.
--lint passes with no complaints.
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by Nix <ni...@esperi.org.uk>.
On 25 Jun 2007, Theo Van Dinter stated:
> I've asked that he open a bugzilla ticket asking for 3.3 updates, and in the
> mean time reverting to a non-development is a good idea for a non-testing
> environment.
The same thing is true if you install from the 3.2-branch, or at least
it was in the 3.2.0 days: I was running 3.2.1 according to the version,
and there were no updates available for that.
It's quite likely that people will run releases from the stable branch
and expect to be able to get updates for it: the stable branch is
supposed to be, well, stable, after all.
(I'm running 3.2.1 and keeping an eye on the branch for critical fixes
to backport now, instead, but it's hardly ideal.)
--
`... in the sense that dragons logically follow evolution so they would
be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
furiously
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
felicity has faster fingers.
He beat me to the puchline.
Thanks
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by Theo Van Dinter <fe...@apache.org>.
On Mon, Jun 25, 2007 at 02:47:03PM -0500, Richard Frovarp wrote:
> Your problem isn't anything to do with the datadir. It has to do with
> the fact sa-update isn't running. Furthermore, sa-update thinks your
> running version 3.3.0. The newest stable release is 3.2.1, so it looks
> to me that you're running a dev version.
Yeah, we've been chatting about this in IRC. Basically, 3.3 has no updates
available at all, but he's downloading updates from SARE, etc. There's a
requirement that if you want the default rules, and you use sa-update, then
you have to use the updates.spamassassin.org channel, and since it doesn't
exist for 3.3, the install is in violation of that.
I've asked that he open a bugzilla ticket asking for 3.3 updates, and in the
mean time reverting to a non-development is a good idea for a non-testing
environment.
--
Randomly Selected Tagline:
"And, although some really nasty mind-games were played, no entities were
physically harmed during the making of this interactive entertainment
(except for the botched special-effect on the bunny rabbit that went so
horribly wrong and really bummed everyone out, no thanks to Mr. Boomer)."
- From the 7th Guest
Re: SA fails to search specified DATADIR for Distribution files
Posted by Richard Frovarp <Ri...@sendit.nodak.edu>.
JT DeLys wrote:
>
> Checking where the updates SHOULD be,
>
> ls -d
> /usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
>
> returns,
>
> /usr/local/bin/ls: cannot access
> /usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org:
> No such file or directory
>
> Well, that's a problem.
>
> looking at output of 'sa-update --debug',
>
> ...
> [18621] dbg: logger: adding facilities: all
> [18621] dbg: logger: logging level is DBG
> [18621] dbg: generic: SpamAssassin version 3.3.0-r543787
> [18621] dbg: config: score set 0 chosen.
> [18621] dbg: dns: is Net::DNS::Resolver available? yes
> [18621] dbg: dns: Net::DNS version: 0.60
> [18621] dbg: generic: sa-update version svn540382
> [18621] dbg: generic: using update directory:
> /usr/local/etc/spamassassin/Updates/3.003000
> ...
> [18621] dbg: channel: reading in channelfile
> /usr/local/etc/spamassassin/sa-update-channels.conf
> [18621] dbg: channel: adding updates.spamassassin.org
> <http://updates.spamassassin.org>
> [18621] dbg: channel: attempting channel updates.spamassassin.org
> <http://updates.spamassassin.org>
> [18621] dbg: channel: update directory
> /usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
> [18621] dbg: channel: channel cf file
> /usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.cf
> [18621] dbg: channel: channel pre file
> /usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.pre
> [18621] dbg: dns: query failed: 0.3.3.updates.spamassassin.org
> <http://0.3.3.updates.spamassassin.org> => NXDOMAIN
> [18621] dbg: channel: no updates available, skipping channel
> [18621] dbg: diag: updates complete, exiting with code 1
>
> I notice the dns query FAIL.
>
> checking the man page,
>
> "The default channel is
> updates.spamassassin.org <http://updates.spamassassin.org>,
> which has updated rules since the previous
> release."
>
>
> Net::DNS, which I think is responsble for those queries, /is/ working,
> as updates of other channels are working.
>
> Ideas?
Your problem isn't anything to do with the datadir. It has to do with
the fact sa-update isn't running. Furthermore, sa-update thinks your
running version 3.3.0. The newest stable release is 3.2.1, so it looks
to me that you're running a dev version.
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
> FYI -- trunk/3.3.0 updates are now working.
Thanks :-)
Although, I think I'll also take earlier advice & 'stay here' (a stable &
working 3.2.x) for a bit.
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
JT DeLys wrote:
>> If you were to use 3.2 it would work.
>
> I've already 'downgraded' to svn 3.2-branch, and you're right - It works!
>
>> Reverting the changes that broke 3.3 updates is on my list of things
>> to do.
>
> Great. Thanks.
FYI -- trunk/3.3.0 updates are now working.
Daryl
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
> If you were to use 3.2 it would work.
I've already 'downgraded' to svn 3.2-branch, and you're right - It works!
> Reverting the changes that broke 3.3 updates is on my list of things to do.
Great. Thanks.
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
JT DeLys wrote:
> Everything relevant ONLY seems to be in Distribution/ -- nothing in
> Updates/.
The problem is that there currently is no update for the SVN version
(3.3.0). If you were to use 3.2 it would work.
Reverting the changes that broke 3.3 updates is on my list of things to do.
Daryl
Re: SA fails to search specified DATADIR for Distribution files
Posted by JT DeLys <jt...@gmail.com>.
> I'm not sure what the problem is you are attempting to demonstrate...
> the update from updates.spamassassin.org contains everything you need
> from the default DATADIR.
The issue parallels a discussion I've been having in IRC.
The problem is that I'm not seeing any SPF checks being done, despite the
Plugin being enabled AND the rules/scores being defined in "50_scores.cf".
'felicity' suggested that's because the rules/scores are not being /seen/ --
and, checking with LINT, there's, indeed, no trace of those rules -- or
ANYTHING from the Distribution files being found/read.
Left to my own devices, I'm trying to figure out why not.
Again, I've specified where the Distribution files are -- why are they not
being found/read?
You state, "the update from updates.spamassassin.org contains everything you
need from the default DATADIR."
Checking my setup/install,
cd /usr/local/etc/spamassassin
grep -i spf `grep -rlni spf .` | grep -i score
./Distribution/50_scores.cf:score RCVD_IN_IADB_SPF 0 -0.001 0 -0.078
./Distribution/50_scores.cf:ifplugin Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:score USER_IN_SPF_WHITELIST -100.000
./Distribution/50_scores.cf:score USER_IN_DEF_SPF_WL -7.500
./Distribution/50_scores.cf:score ENV_AND_HDR_SPF_MATCH -7.500
./Distribution/50_scores.cf:endif # Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:#
SPF
./Distribution/50_scores.cf:# Note that the benefit for a valid SPF record
is deliberately minimal; it's
./Distribution/50_scores.cf:# likely that more spammers would quickly move
to setting valid SPF records
./Distribution/50_scores.cf:ifplugin Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:score SPF_PASS -0.001
./Distribution/50_scores.cf:score SPF_HELO_PASS -0.001
./Distribution/50_scores.cf:score SPF_FAIL 2.600 0.992 1.669 0.693
./Distribution/50_scores.cf:score SPF_HELO_FAIL 2.298 0.365 0.540 0.001
./Distribution/50_scores.cf:score SPF_HELO_NEUTRAL 2.231 2.000 0.744 0.576
./Distribution/50_scores.cf:score SPF_HELO_SOFTFAIL 2.599 1.533 1.427 0.841
./Distribution/50_scores.cf:score SPF_NEUTRAL 2.199 1.210 0.756 0.686
./Distribution/50_scores.cf:score SPF_SOFTFAIL 2.301 0.654 0.698 0.596
./Distribution/50_scores.cf:endif # Mail::SpamAssassin::Plugin::SPF
Everything relevant ONLY seems to be in Distribution/ -- nothing in
Updates/.
Checking where the updates SHOULD be,
ls -d
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
returns,
/usr/local/bin/ls: cannot access
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org: No
such file or directory
Well, that's a problem.
looking at output of 'sa-update --debug',
...
[18621] dbg: logger: adding facilities: all
[18621] dbg: logger: logging level is DBG
[18621] dbg: generic: SpamAssassin version 3.3.0-r543787
[18621] dbg: config: score set 0 chosen.
[18621] dbg: dns: is Net::DNS::Resolver available? yes
[18621] dbg: dns: Net::DNS version: 0.60
[18621] dbg: generic: sa-update version svn540382
[18621] dbg: generic: using update directory:
/usr/local/etc/spamassassin/Updates/3.003000
...
[18621] dbg: channel: reading in channelfile /usr/local/etc/spamassassin/sa-
update-channels.conf
[18621] dbg: channel: adding updates.spamassassin.org
[18621] dbg: channel: attempting channel updates.spamassassin.org
[18621] dbg: channel: update directory
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
[18621] dbg: channel: channel cf file
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.cf
[18621] dbg: channel: channel pre file
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.pre
[18621] dbg: dns: query failed: 0.3.3.updates.spamassassin.org => NXDOMAIN
[18621] dbg: channel: no updates available, skipping channel
[18621] dbg: diag: updates complete, exiting with code 1
I notice the dns query FAIL.
checking the man page,
"The default channel is
updates.spamassassin.org, which has updated rules since the previous
release."
Net::DNS, which I think is responsble for those queries, /is/ working, as
updates of other channels are working.
Ideas?
--
Thanks,
JTDeLys
Re: SA fails to search specified DATADIR for Distribution files
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
JT DeLys wrote:
> It seems to me that, according to,
>
> " Default configuration data is loaded from the first existing
> directory
> in:
> "
>
> the Default config data is NOT picked up from the first INSTANCE of a
> given .cf in the search hierarchy, just from the first DIR listed there.
Yeah, as you quoted, the docs say "first existing directory" and not
filename. The software, perhaps coincidentally, works the same way.
I'm not sure what the problem is you are attempting to demonstrate...
the update from updates.spamassassin.org contains everything you need
from the default DATADIR.
Daryl