You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spamassassin.apache.org by si...@apache.org on 2022/12/17 11:38:45 UTC
svn commit: r1906055 [16/18] - in /spamassassin/site/full/4.0.x: ./ doc/
Added: spamassassin/site/full/4.0.x/doc/sa-learn.html
URL: http://svn.apache.org/viewvc/spamassassin/site/full/4.0.x/doc/sa-learn.html?rev=1906055&view=auto
==============================================================================
--- spamassassin/site/full/4.0.x/doc/sa-learn.html (added)
+++ spamassassin/site/full/4.0.x/doc/sa-learn.html Sat Dec 17 11:38:43 2022
@@ -0,0 +1,711 @@
+<?xml version="1.0" ?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+<title>sa-learn - train SpamAssassin's Bayesian classifier</title>
+<meta http-equiv="content-type" content="text/html; charset=utf-8" />
+<link rev="made" href="mailto:root@localhost" />
+</head>
+
+<body>
+
+
+
+<ul id="index">
+ <li><a href="#NAME">NAME</a></li>
+ <li><a href="#SYNOPSIS">SYNOPSIS</a></li>
+ <li><a href="#DESCRIPTION">DESCRIPTION</a></li>
+ <li><a href="#OPTIONS">OPTIONS</a></li>
+ <li><a href="#MIGRATION">MIGRATION</a></li>
+ <li><a href="#INTRODUCTION-TO-BAYESIAN-FILTERING">INTRODUCTION TO BAYESIAN FILTERING</a></li>
+ <li><a href="#GETTING-STARTED">GETTING STARTED</a></li>
+ <li><a href="#EFFECTIVE-TRAINING">EFFECTIVE TRAINING</a></li>
+ <li><a href="#FILES">FILES</a></li>
+ <li><a href="#EXPIRATION">EXPIRATION</a>
+ <ul>
+ <li><a href="#EXPIRE-LOGIC">EXPIRE LOGIC</a></li>
+ <li><a href="#ESTIMATION-PASS-LOGIC">ESTIMATION PASS LOGIC</a></li>
+ <li><a href="#EXPIRY-RELATED-CONFIGURATION-SETTINGS">EXPIRY RELATED CONFIGURATION SETTINGS</a></li>
+ </ul>
+ </li>
+ <li><a href="#INSTALLATION">INSTALLATION</a></li>
+ <li><a href="#SEE-ALSO">SEE ALSO</a></li>
+ <li><a href="#PREREQUISITES">PREREQUISITES</a></li>
+ <li><a href="#AUTHORS">AUTHORS</a></li>
+</ul>
+
+<h1 id="NAME">NAME</h1>
+
+<p>sa-learn - train SpamAssassin's Bayesian classifier</p>
+
+<h1 id="SYNOPSIS">SYNOPSIS</h1>
+
+<p><b>sa-learn</b> [options] [file]...</p>
+
+<p><b>sa-learn</b> [options] --dump [ all | data | magic ]</p>
+
+<p>Options:</p>
+
+<pre><code>--ham Learn the following messages as ham (non-spam)
+--spam Learn the following messages as spam
+--forget Forget the following messages
+--use-ignores Use bayes_ignore_from and bayes_ignore_to
+--sync Synchronize the database and the journal if needed
+--force-expire Force a database sync and expiry run
+--dbpath <path> Allows commandline override (in bayes_path form)
+ for where to read the Bayes DB from
+--dump [all|data|magic] Display the contents of the Bayes database
+ Takes optional argument for what to display
+ --regexp <re> For dump only, specifies which tokens to
+ dump based on a regular expression.
+-f file, --folders=file Read list of files/directories from file
+--dir Ignored; historical compatibility
+--file Ignored; historical compatibility
+--mbox Input sources are in mbox format
+--mbx Input sources are in mbx format
+--max-size <b> Skip messages larger than b bytes;
+ defaults to 500 KB, 0 implies no limit
+--showdots Show progress using dots
+--progress Show progress using progress bar
+--no-sync Skip synchronizing the database and journal
+ after learning
+-L, --local Operate locally, no network accesses. Use
+ of this is recommended, see documentation.
+--import Migrate data from older version/non DB_File
+ based databases
+--clear Wipe out existing database
+--backup Backup, to STDOUT, existing database
+--restore <filename> Restore a database from filename
+-u username, --username=username
+ Override username taken from the runtime
+ environment, used with SQL
+-C path, --configpath=path, --config-file=path
+ Path to standard configuration dir
+-p prefs, --prefspath=file, --prefs-file=file
+ Set user preferences file
+--siteconfigpath=path Path for site configs
+ (default: /usr/local/etc/mail/spamassassin)
+--cf='config line' Additional line of configuration
+-D, --debug [area,...] Print debugging messages
+-V, --version Print version
+-h, --help Print usage message</code></pre>
+
+<h1 id="DESCRIPTION">DESCRIPTION</h1>
+
+<p>Given a typical selection of your incoming mail classified as spam or ham (non-spam), this tool will feed each mail to SpamAssassin, allowing it to 'learn' what signs are likely to mean spam, and which are likely to mean ham.</p>
+
+<p>Simply run this command once for each of your mail folders, and it will ''learn'' from the mail therein.</p>
+
+<p>Note that csh-style <i>globbing</i> in the mail folder names is supported; in other words, listing a folder name as <code>*</code> will scan every folder that matches. See <code>Mail::SpamAssassin::ArchiveIterator</code> for more details.</p>
+
+<p>If you are using mail boxes in format other than maildir you should use the <b>--mbox</b> or <b>--mbx</b> parameters.</p>
+
+<p>Files compressed with gzip/bzip2/xz/lz4/lzip/lzo are uncompressed automatically. See <code>Mail::SpamAssassin::ArchiveIterator</code> for more details.</p>
+
+<p>SpamAssassin remembers which mail messages it has learnt already, and will not re-learn those messages again, unless you use the <b>--forget</b> option. Messages learnt as spam will have SpamAssassin markup removed, on the fly.</p>
+
+<p>If you make a mistake and scan a mail as ham when it is spam, or vice versa, simply rerun this command with the correct classification, and the mistake will be corrected. SpamAssassin will automatically 'forget' the previous indications.</p>
+
+<p>Users of <code>spamd</code> who wish to perform training remotely, over a network, should investigate the <code>spamc -L</code> switch.</p>
+
+<h1 id="OPTIONS">OPTIONS</h1>
+
+<dl>
+
+<dt id="ham"><b>--ham</b></dt>
+<dd>
+
+<p>Learn the input message(s) in the files following the option as ham. If you have previously learnt any of the messages as spam, SpamAssassin will forget them first, then re-learn them as ham. Alternatively, if you have previously learnt them as ham, it'll skip them this time around. If the messages have already been filtered through SpamAssassin, the learner will ignore any modifications SpamAssassin may have made.</p>
+
+</dd>
+<dt id="spam"><b>--spam</b></dt>
+<dd>
+
+<p>Learn the input message(s) in the files following the option as spam. If you have previously learnt any of the messages as ham, SpamAssassin will forget them first, then re-learn them as spam. Alternatively, if you have previously learnt them as spam, it'll skip them this time around. If the messages have already been filtered through SpamAssassin, the learner will ignore any modifications SpamAssassin may havemmade.</p>
+
+</dd>
+<dt id="folders-filename--f-filename"><b>--folders</b>=<i>filename</i>, <b>-f</b> <i>filename</i></dt>
+<dd>
+
+<p>sa-learn will read in the list of folders from the specified file, one folder per line in the file. If the folder is prefixed with <code>ham:type:</code> or <code>spam:type:</code>, sa-learn will learn that folder appropriately, otherwise the folders will be assumed to be of the type specified by <b>--ham</b> or <b>--spam</b>.</p>
+
+<p><code>type</code> above is optional, but is the same as the standard for ArchiveIterator: mbox, mbx, dir, file, or detect (the default if not specified).</p>
+
+</dd>
+<dt id="mbox"><b>--mbox</b></dt>
+<dd>
+
+<p>sa-learn will read in the file(s) containing the emails to be learned, and will process them in mbox format (one or more emails per file).</p>
+
+</dd>
+<dt id="mbx"><b>--mbx</b></dt>
+<dd>
+
+<p>sa-learn will read in the file(s) containing the emails to be learned, and will process them in mbx format (one or more emails per file).</p>
+
+</dd>
+<dt id="use-ignores"><b>--use-ignores</b></dt>
+<dd>
+
+<p>Don't learn the message if a from address matches configuration file item <code>bayes_ignore_from</code> or a to address matches <code>bayes_ignore_to</code>. The option might be used when learning from a large file of messages from which the hammy spam messages or spammy ham messages have not been removed.</p>
+
+</dd>
+<dt id="sync"><b>--sync</b></dt>
+<dd>
+
+<p>Synchronize the journal and databases. Upon successfully syncing the database with the entries in the journal, the journal file is removed.</p>
+
+</dd>
+<dt id="force-expire"><b>--force-expire</b></dt>
+<dd>
+
+<p>Forces an expiry attempt, regardless of whether it may be necessary or not. Note: This doesn't mean any tokens will actually expire. Please see the EXPIRATION section below.</p>
+
+<p>Note: <code>--force-expire</code> also causes the journal data to be synchronized into the Bayes databases.</p>
+
+</dd>
+<dt id="forget"><b>--forget</b></dt>
+<dd>
+
+<p>Forget the input message(s) in the files following the option as previously learnt.</p>
+
+</dd>
+<dt id="dbpath"><b>--dbpath</b></dt>
+<dd>
+
+<p>Allows a commandline override of the <i>bayes_path</i> configuration option.</p>
+
+</dd>
+<dt id="dump-option"><b>--dump</b> <i>option</i></dt>
+<dd>
+
+<p>Display the contents of the Bayes database. Without an option or with the <i>all</i> option, all magic tokens and data tokens will be displayed. <i>magic</i> will only display magic tokens, and <i>data</i> will only display the data tokens.</p>
+
+<p>Can also use the <b>--regexp</b> <i>RE</i> option to specify which tokens to display based on a regular expression.</p>
+
+</dd>
+<dt id="clear"><b>--clear</b></dt>
+<dd>
+
+<p>Clear an existing Bayes database by removing all traces of the database.</p>
+
+<p>WARNING: This is destructive and should be used with care.</p>
+
+</dd>
+<dt id="backup"><b>--backup</b></dt>
+<dd>
+
+<p>Performs a dump of the Bayes database in machine/human readable format.</p>
+
+<p>The dump will include token and seen data. It is suitable for input back into the --restore command.</p>
+
+</dd>
+<dt id="restore-filename"><b>--restore</b>=<i>filename</i></dt>
+<dd>
+
+<p>Performs a restore of the Bayes database defined by <i>filename</i>.</p>
+
+<p>WARNING: This is a destructive operation, previous Bayes data will be wiped out.</p>
+
+</dd>
+<dt id="h---help"><b>-h</b>, <b>--help</b></dt>
+<dd>
+
+<p>Print help message and exit.</p>
+
+</dd>
+<dt id="u-username---username-username"><b>-u</b> <i>username</i>, <b>--username</b>=<i>username</i></dt>
+<dd>
+
+<p>If specified this username will override the username taken from the runtime environment. You can use this option to specify users in a virtual user configuration when using SQL as the Bayes backend.</p>
+
+<p>NOTE: This option will not change to the given <i>username</i>, it will only attempt to act on behalf of that user. Because of this you will need to have proper permissions to be able to change files owned by <i>username</i>. In the case of SQL this generally is not a problem.</p>
+
+</dd>
+<dt id="C-path---configpath-path---config-file-path"><b>-C</b> <i>path</i>, <b>--configpath</b>=<i>path</i>, <b>--config-file</b>=<i>path</i></dt>
+<dd>
+
+<p>Use the specified path for locating the distributed configuration files. Ignore the default directories (usually <code>/usr/share/spamassassin</code> or similar).</p>
+
+</dd>
+<dt id="siteconfigpath-path"><b>--siteconfigpath</b>=<i>path</i></dt>
+<dd>
+
+<p>Use the specified path for locating site-specific configuration files. Ignore the default directories (usually <code>/etc/mail/spamassassin</code> or similar).</p>
+
+</dd>
+<dt id="cf-config-line"><b>--cf='config line'</b></dt>
+<dd>
+
+<p>Add additional lines of configuration directly from the command-line, parsed after the configuration files are read. Multiple <b>--cf</b> arguments can be used, and each will be considered a separate line of configuration.</p>
+
+</dd>
+<dt id="p-prefs---prefspath-prefs---prefs-file-prefs"><b>-p</b> <i>prefs</i>, <b>--prefspath</b>=<i>prefs</i>, <b>--prefs-file</b>=<i>prefs</i></dt>
+<dd>
+
+<p>Read user score preferences from <i>prefs</i> (usually <code>$HOME/.spamassassin/user_prefs</code>).</p>
+
+</dd>
+<dt id="progress"><b>--progress</b></dt>
+<dd>
+
+<p>Prints a progress bar (to STDERR) showing the current progress. In the case where no valid terminal is found this option will behave very much like the --showdots option.</p>
+
+</dd>
+<dt id="D-area-...---debug-area"><b>-D</b> [<i>area,...</i>], <b>--debug</b> [<i>area,...</i>]</dt>
+<dd>
+
+<p>Produce debugging output. If no areas are listed, all debugging information is printed. Diagnostic output can also be enabled for each area individually; <i>area</i> is the area of the code to instrument. For example, to produce diagnostic output on bayes, learn, and dns, use:</p>
+
+<pre><code>spamassassin -D bayes,learn,dns</code></pre>
+
+<p>Use an empty string (-D '') to indicate no areas when the next item on the command line is a path, to prevent the path from being parsed as an area.</p>
+
+<p>For more information about which areas (also known as channels) are available, please see the documentation at:</p>
+
+<pre><code>C<https://wiki.apache.org/spamassassin/DebugChannels></code></pre>
+
+<p>Higher priority informational messages that are suitable for logging in normal circumstances are available with an area of "info".</p>
+
+</dd>
+<dt id="no-sync"><b>--no-sync</b></dt>
+<dd>
+
+<p>Skip the slow synchronization step which normally takes place after changing database entries. If you plan to learn from many folders in a batch, or to learn many individual messages one-by-one, it is faster to use this switch and run <code>sa-learn --sync</code> once all the folders have been scanned.</p>
+
+<p>Clarification: The state of <i>--no-sync</i> overrides the <i>bayes_learn_to_journal</i> configuration option. If not specified, sa-learn will learn to the database directly. If specified, sa-learn will learn to the journal file.</p>
+
+<p>Note: <i>--sync</i> and <i>--no-sync</i> can be specified on the same commandline, which is slightly confusing. In this case, the <i>--no-sync</i> option is ignored since there is no learn operation.</p>
+
+</dd>
+<dt id="L---local"><b>-L</b>, <b>--local</b></dt>
+<dd>
+
+<p>Do not perform any network accesses while learning details about the mail messages. This should be normally used, as there really isn't anything Bayes can learn from network lookup results. Official SpamAssassin plugins do not currently do any network lookups when learning, but it's possible that third party ones might.</p>
+
+</dd>
+<dt id="import"><b>--import</b></dt>
+<dd>
+
+<p>If you previously used SpamAssassin's Bayesian learner without the <code>DB_File</code> module installed, it will have created files in other formats, such as <code>GDBM_File</code>, <code>NDBM_File</code>, or <code>SDBM_File</code>. This switch allows you to migrate that old data into the <code>DB_File</code> format. It will overwrite any data currently in the <code>DB_File</code>.</p>
+
+<p>Can also be used with the <b>--dbpath</b> <i>path</i> option to specify the location of the Bayes files to use.</p>
+
+</dd>
+</dl>
+
+<h1 id="MIGRATION">MIGRATION</h1>
+
+<p>There are now multiple backend storage modules available for storing user's bayesian data. As such you might want to migrate from one backend to another. Here is a simple procedure for migrating from one backend to another.</p>
+
+<p>Note that if you have individual user databases you will have to perform a similar procedure for each one of them.</p>
+
+<dl>
+
+<dt id="sa-learn---sync">sa-learn --sync</dt>
+<dd>
+
+<p>This will sync any outstanding journal entries</p>
+
+</dd>
+<dt id="sa-learn---backup-backup.txt">sa-learn --backup > backup.txt</dt>
+<dd>
+
+<p>This will save all your Bayes data to a plain text file.</p>
+
+</dd>
+<dt id="sa-learn---clear">sa-learn --clear</dt>
+<dd>
+
+<p>This is optional, but good to do to clear out the old database.</p>
+
+</dd>
+<dt id="Repeat">Repeat!</dt>
+<dd>
+
+<p>At this point, if you have multiple databases, you should perform the procedure above for each of them. (i.e. each user's database needs to be backed up before continuing.)</p>
+
+</dd>
+<dt id="Switch-backends">Switch backends</dt>
+<dd>
+
+<p>Once you have backed up all databases you can update your configuration for the new database backend. This will involve at least the bayes_store_module config option and may involve some additional config options depending on what is required by the module. (For example, you may need to configure an SQL database.)</p>
+
+</dd>
+<dt id="sa-learn---restore-backup.txt">sa-learn --restore backup.txt</dt>
+<dd>
+
+<p>Again, you need to do this for every database.</p>
+
+</dd>
+</dl>
+
+<p>If you are migrating to SQL you can make use of the -u <username> option in sa-learn to populate each user's database. Otherwise, you must run sa-learn as the user who database you are restoring.</p>
+
+<h1 id="INTRODUCTION-TO-BAYESIAN-FILTERING">INTRODUCTION TO BAYESIAN FILTERING</h1>
+
+<p>(Thanks to Michael Bell for this section!)</p>
+
+<p>For a more lengthy description of how this works, go to http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably readable, even if statistics make me break out in hives.</p>
+
+<p>The short semi-inaccurate version: Given training, a spam heuristics engine can take the most "spammy" and "hammy" words and apply probabilistic analysis. Furthermore, once given a basis for the analysis, the engine can continue to learn iteratively by applying both the non-Bayesian and Bayesian rulesets together to create evolving "intelligence".</p>
+
+<p>SpamAssassin 2.50 and later supports Bayesian spam analysis, in the form of the BAYES rules. This is a new feature, quite powerful, and is disabled until enough messages have been learnt.</p>
+
+<p>The pros of Bayesian spam analysis:</p>
+
+<dl>
+
+<dt id="Can-greatly-reduce-false-positives-and-false-negatives">Can greatly reduce false positives and false negatives.</dt>
+<dd>
+
+<p>It learns from your mail, so it is tailored to your unique e-mail flow.</p>
+
+</dd>
+<dt id="Once-it-starts-learning-it-can-continue-to-learn-from-SpamAssassin-and-improve-over-time">Once it starts learning, it can continue to learn from SpamAssassin and improve over time.</dt>
+<dd>
+
+</dd>
+</dl>
+
+<p>And the cons:</p>
+
+<dl>
+
+<dt id="A-decent-number-of-messages-are-required-before-results-are-useful-for-ham-spam-determination">A decent number of messages are required before results are useful for ham/spam determination.</dt>
+<dd>
+
+</dd>
+<dt id="Its-hard-to-explain-why-a-message-is-or-isnt-marked-as-spam">It's hard to explain why a message is or isn't marked as spam.</dt>
+<dd>
+
+<p>i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy to understand. If it generates a false positive or false negative, it is fairly easy to understand why.</p>
+
+<p>With Bayesian analysis, it's all probabilities - "because the past says it is likely as this falls into a probabilistic distribution common to past spam in your systems". Tell that to your users! Tell that to the client when he asks "what can I do to change this". (By the way, the answer in this case is "use welcomelisting".)</p>
+
+</dd>
+<dt id="It-will-take-disk-space-and-memory">It will take disk space and memory.</dt>
+<dd>
+
+<p>The databases it maintains take quite a lot of resources to store and use.</p>
+
+</dd>
+</dl>
+
+<h1 id="GETTING-STARTED">GETTING STARTED</h1>
+
+<p>Still interested? Ok, here's the guidelines for getting this working.</p>
+
+<p>First a high-level overview:</p>
+
+<dl>
+
+<dt id="Build-a-significant-sample-of-both-ham-and-spam">Build a significant sample of both ham and spam.</dt>
+<dd>
+
+<p>I suggest several thousand of each, placed in SPAM and HAM directories or mailboxes. Yes, you MUST hand-sort this - otherwise the results won't be much better than SpamAssassin on its own. Verify the spamminess/haminess of EVERY message. You're urged to avoid using a publicly available corpus (sample) - this must be taken from YOUR mail server, if it is to be statistically useful. Otherwise, the results may be pretty skewed.</p>
+
+</dd>
+<dt id="Use-this-tool-to-teach-SpamAssassin-about-these-samples-like-so">Use this tool to teach SpamAssassin about these samples, like so:</dt>
+<dd>
+
+<pre><code>sa-learn --spam /path/to/spam/folder
+sa-learn --ham /path/to/ham/folder
+sa-learn --ham hampath1 hampath2 --spam spampath1 spampath2
+...</code></pre>
+
+<p>Let SpamAssassin proceed, learning stuff. When it finds ham and spam it will add the "interesting tokens" to the database.</p>
+
+</dd>
+<dt id="If-you-need-SpamAssassin-to-forget-about-specific-messages-use-the---forget-option">If you need SpamAssassin to forget about specific messages, use the <b>--forget</b> option.</dt>
+<dd>
+
+<p>This can be applied to either ham or spam that has run through the <b>sa-learn</b> processes. It's a bit of a hammer, really, lowering the weighting of the specific tokens in that message (only if that message has been processed before).</p>
+
+</dd>
+<dt id="Learning-from-single-messages-uses-a-command-like-this">Learning from single messages uses a command like this:</dt>
+<dd>
+
+<pre><code>sa-learn --ham --no-sync mailmessage</code></pre>
+
+<p>This is handy for binding to a key in your mail user agent. It's very fast, as all the time-consuming stuff is deferred until you run with the <code>--sync</code> option.</p>
+
+</dd>
+<dt id="Autolearning-is-enabled-by-default">Autolearning is enabled by default</dt>
+<dd>
+
+<p>If you don't have a corpus of mail saved to learn, you can let SpamAssassin automatically learn the mail that you receive. If you are autolearning from scratch, the amount of mail you receive will determine how long until the BAYES_* rules are activated.</p>
+
+</dd>
+</dl>
+
+<h1 id="EFFECTIVE-TRAINING">EFFECTIVE TRAINING</h1>
+
+<p>Learning filters require training to be effective. If you don't train them, they won't work. In addition, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy.</p>
+
+<p>You need to train with both spam <i>and</i> ham mails. One type of mail alone will not have any effect.</p>
+
+<p>Note that if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.)</p>
+
+<p>If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run <code>spamassassin -d</code> over it in advance.</p>
+
+<p>Another thing to be aware of, is that typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests.</p>
+
+<p>Be careful that you train from the same source -- for example, if you train on old spam, but new ham mail, then the classifier will think that a mail with an old date stamp is likely to be spam.</p>
+
+<p>It's also worth noting that training with a very small quantity of ham, will produce atrocious results. You should aim to train with at least the same amount (or more if possible!) of ham data than spam.</p>
+
+<p>On an on-going basis, it is best to keep training the filter to make sure it has fresh data to work from. There are various ways to do this:</p>
+
+<dl>
+
+<dt id="Supervised-learning">1. Supervised learning</dt>
+<dd>
+
+<p>This means keeping a copy of all or most of your mail, separated into spam and ham piles, and periodically re-training using those. It produces the best results, but requires more work from you, the user.</p>
+
+<p>(An easy way to do this, by the way, is to create a new folder for 'deleted' messages, and instead of deleting them from other folders, simply move them in there instead. Then keep all spam in a separate folder and never delete it. As long as you remember to move misclassified mails into the correct folder set, it is easy enough to keep up to date.)</p>
+
+</dd>
+<dt id="Unsupervised-learning-from-Bayesian-classification">2. Unsupervised learning from Bayesian classification</dt>
+<dd>
+
+<p>Another way to train is to chain the results of the Bayesian classifier back into the training, so it reinforces its own decisions. This is only safe if you then retrain it based on any errors you discover.</p>
+
+<p>SpamAssassin does not support this method, due to experimental results which strongly indicate that it does not work well, and since Bayes is only one part of the resulting score presented to the user (while Bayes may have made the wrong decision about a mail, it may have been overridden by another system).</p>
+
+</dd>
+<dt id="Unsupervised-learning-from-SpamAssassin-rules">3. Unsupervised learning from SpamAssassin rules</dt>
+<dd>
+
+<p>Also called 'auto-learning' in SpamAssassin. Based on statistical analysis of the SpamAssassin success rates, we can automatically train the Bayesian database with a certain degree of confidence that our training data is accurate.</p>
+
+<p>It should be supplemented with some supervised training in addition, if possible.</p>
+
+<p>This is the default, but can be turned off by setting the SpamAssassin configuration parameter <code>bayes_auto_learn</code> to 0.</p>
+
+</dd>
+<dt id="Mistake-based-training">4. Mistake-based training</dt>
+<dd>
+
+<p>This means training on a small number of mails, then only training on messages that SpamAssassin classifies incorrectly. This works, but it takes longer to get it right than a full training session would.</p>
+
+</dd>
+</dl>
+
+<h1 id="FILES">FILES</h1>
+
+<p><b>sa-learn</b> and the other parts of SpamAssassin's Bayesian learner, use a set of persistent database files to store the learnt tokens, as follows.</p>
+
+<dl>
+
+<dt id="bayes_toks">bayes_toks</dt>
+<dd>
+
+<p>The database of tokens, containing the tokens learnt, their count of occurrences in ham and spam, and the timestamp when the token was last seen in a message.</p>
+
+<p>This database also contains some 'magic' tokens, as follows: the version number of the database, the number of ham and spam messages learnt, the number of tokens in the database, and timestamps of: the last journal sync, the last expiry run, the last expiry token reduction count, the last expiry timestamp delta, the oldest token timestamp in the database, and the newest token timestamp in the database.</p>
+
+<p>This is a database file, using <code>DB_File</code>. The database 'version number' is 0 for databases from 2.5x, 1 for databases from certain 2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later releases.</p>
+
+</dd>
+<dt id="bayes_seen">bayes_seen</dt>
+<dd>
+
+<p>A map of Message-Id and some data from headers and body to what that message was learnt as. This is used so that SpamAssassin can avoid re-learning a message it has already seen, and so it can reverse the training if you later decide that message was learnt incorrectly.</p>
+
+<p>This is a database file, using <code>DB_File</code>.</p>
+
+</dd>
+<dt id="bayes_journal">bayes_journal</dt>
+<dd>
+
+<p>While SpamAssassin is scanning mails, it needs to track which tokens it uses in its calculations. To avoid the contention of having each SpamAssassin process attempting to gain write access to the Bayes DB, the token timestamps are written to a 'journal' file which will later (either automatically or via <code>sa-learn --sync</code>) be used to synchronize the Bayes DB.</p>
+
+<p>Also, through the use of <code>bayes_learn_to_journal</code>, or when using the <code>--no-sync</code> option with sa-learn, the actual learning data will take be placed into the journal for later synchronization. This is typically useful for high-traffic sites to avoid the same contention as stated above.</p>
+
+</dd>
+</dl>
+
+<h1 id="EXPIRATION">EXPIRATION</h1>
+
+<p>Since SpamAssassin can auto-learn messages, the Bayes database files could increase perpetually until they fill your disk. To control this, SpamAssassin performs journal synchronization and bayes expiration periodically when certain criteria (listed below) are met.</p>
+
+<p>SpamAssassin can sync the journal and expire the DB tokens either manually or opportunistically. A journal sync is due if <i>--sync</i> is passed to sa-learn (manual), or if the following is true (opportunistic):</p>
+
+<dl>
+
+<dt id="bayes_journal_max_size-does-not-equal-0-means-dont-sync">- bayes_journal_max_size does not equal 0 (means don't sync)</dt>
+<dd>
+
+</dd>
+<dt id="the-journal-file-exists">- the journal file exists</dt>
+<dd>
+
+</dd>
+</dl>
+
+<p>and either:</p>
+
+<dl>
+
+<dt id="the-journal-file-has-a-size-greater-than-bayes_journal_max_size">- the journal file has a size greater than bayes_journal_max_size</dt>
+<dd>
+
+</dd>
+</dl>
+
+<p>or</p>
+
+<dl>
+
+<dt id="a-journal-sync-has-previously-occurred-and-at-least-1-day-has-passed-since-that-sync">- a journal sync has previously occurred, and at least 1 day has passed since that sync</dt>
+<dd>
+
+</dd>
+</dl>
+
+<p>Expiry is due if <i>--force-expire</i> is passed to sa-learn (manual), or if all of the following are true (opportunistic):</p>
+
+<dl>
+
+<dt id="the-last-expire-was-attempted-at-least-12hrs-ago">- the last expire was attempted at least 12hrs ago</dt>
+<dd>
+
+</dd>
+<dt id="bayes_auto_expire-does-not-equal-0">- bayes_auto_expire does not equal 0</dt>
+<dd>
+
+</dd>
+<dt id="the-number-of-tokens-in-the-DB-is-100-000">- the number of tokens in the DB is > 100,000</dt>
+<dd>
+
+</dd>
+<dt id="the-number-of-tokens-in-the-DB-is-bayes_expiry_max_db_size">- the number of tokens in the DB is > bayes_expiry_max_db_size</dt>
+<dd>
+
+</dd>
+<dt id="there-is-at-least-a-12-hr-difference-between-the-oldest-and-newest-token-atimes">- there is at least a 12 hr difference between the oldest and newest token atimes</dt>
+<dd>
+
+</dd>
+</dl>
+
+<h2 id="EXPIRE-LOGIC">EXPIRE LOGIC</h2>
+
+<p>If either the manual or opportunistic method causes an expire run to start, here is the logic that is used:</p>
+
+<dl>
+
+<dt id="figure-out-how-many-tokens-to-keep.-take-the-larger-of-either-bayes_expiry_max_db_size-75-or-100-000-tokens.-therefore-the-goal-reduction-is-number-of-tokens---number-of-tokens-to-keep">- figure out how many tokens to keep. take the larger of either bayes_expiry_max_db_size * 75% or 100,000 tokens. therefore, the goal reduction is number of tokens - number of tokens to keep.</dt>
+<dd>
+
+</dd>
+<dt id="if-the-reduction-number-is-1000-tokens-abort-not-worth-the-effort">- if the reduction number is < 1000 tokens, abort (not worth the effort).</dt>
+<dd>
+
+</dd>
+<dt id="if-an-expire-has-been-done-before-guesstimate-the-new-atime-delta-based-on-the-old-atime-delta.-new_atime_delta-old_atime_delta-old_reduction_count-goal">- if an expire has been done before, guesstimate the new atime delta based on the old atime delta. (new_atime_delta = old_atime_delta * old_reduction_count / goal)</dt>
+<dd>
+
+</dd>
+<dt id="if-no-expire-has-been-done-before-or-the-last-expire-looks-weird-do-an-estimation-pass.-The-definition-of-weird-is">- if no expire has been done before, or the last expire looks "weird", do an estimation pass. The definition of "weird" is:</dt>
+<dd>
+
+<dl>
+
+<dt id="last-expire-over-30-days-ago">- last expire over 30 days ago</dt>
+<dd>
+
+</dd>
+<dt id="last-atime-delta-was-12-hrs">- last atime delta was < 12 hrs</dt>
+<dd>
+
+</dd>
+<dt id="last-reduction-count-was-1000-tokens">- last reduction count was < 1000 tokens</dt>
+<dd>
+
+</dd>
+<dt id="estimated-new-atime-delta-is-12-hrs">- estimated new atime delta is < 12 hrs</dt>
+<dd>
+
+</dd>
+<dt id="the-difference-between-the-last-reduction-count-and-the-goal-reduction-count-is-50">- the difference between the last reduction count and the goal reduction count is > 50%</dt>
+<dd>
+
+</dd>
+</dl>
+
+</dd>
+</dl>
+
+<h2 id="ESTIMATION-PASS-LOGIC">ESTIMATION PASS LOGIC</h2>
+
+<p>Go through each of the DB's tokens. Starting at 12hrs, calculate whether or not the token would be expired (based on the difference between the token's atime and the db's newest token atime) and keep the count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs * 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512 (6144hrs, or 256 days).</p>
+
+<p>The larger the delta, the smaller the number of tokens that will be expired. Conversely, the number of tokens goes up as the delta gets smaller. So starting at the largest atime delta, figure out which delta will expire the most tokens without going above the goal expiration count. Use this to choose the atime delta to use, unless one of the following occurs:</p>
+
+<dl>
+
+<dt id="the-largest-atime-smallest-reduction-count-would-expire-too-many-tokens.-this-means-the-learned-tokens-are-mostly-old-and-there-needs-to-be-new-tokens-learned-before-an-expire-can-occur">- the largest atime (smallest reduction count) would expire too many tokens. this means the learned tokens are mostly old and there needs to be new tokens learned before an expire can occur.</dt>
+<dd>
+
+</dd>
+<dt id="all-of-the-atime-choices-result-in-0-tokens-being-removed.-this-means-the-tokens-are-all-newer-than-12-hours-and-there-needs-to-be-new-tokens-learned-before-an-expire-can-occur">- all of the atime choices result in 0 tokens being removed. this means the tokens are all newer than 12 hours and there needs to be new tokens learned before an expire can occur.</dt>
+<dd>
+
+</dd>
+<dt id="the-number-of-tokens-that-would-be-removed-is-1000.-the-benefit-isnt-worth-the-effort.-more-tokens-need-to-be-learned">- the number of tokens that would be removed is < 1000. the benefit isn't worth the effort. more tokens need to be learned.</dt>
+<dd>
+
+</dd>
+</dl>
+
+<p>If the expire run gets past this point, it will continue to the end. A new DB is created since the majority of DB libraries don't shrink the DB file when tokens are removed. So we do the "create new, migrate old to new, remove old, rename new" shuffle.</p>
+
+<h2 id="EXPIRY-RELATED-CONFIGURATION-SETTINGS">EXPIRY RELATED CONFIGURATION SETTINGS</h2>
+
+<dl>
+
+<dt id="bayes_auto_expire-is-used-to-specify-whether-or-not-SpamAssassin-ought-to-opportunistically-attempt-to-expire-the-Bayes-database.-The-default-is-1-yes"><code>bayes_auto_expire</code> is used to specify whether or not SpamAssassin ought to opportunistically attempt to expire the Bayes database. The default is 1 (yes).</dt>
+<dd>
+
+</dd>
+<dt id="bayes_expiry_max_db_size-specifies-both-the-auto-expire-token-count-point-as-well-as-the-resulting-number-of-tokens-after-expiry-as-described-above.-The-default-value-is-150-000-which-is-roughly-equivalent-to-a-6Mb-database-file-if-youre-using-DB_File"><code>bayes_expiry_max_db_size</code> specifies both the auto-expire token count point, as well as the resulting number of tokens after expiry as described above. The default value is 150,000, which is roughly equivalent to a 6Mb database file if you're using DB_File.</dt>
+<dd>
+
+</dd>
+<dt id="bayes_journal_max_size-specifies-how-large-the-Bayes-journal-will-grow-before-it-is-opportunistically-synced.-The-default-value-is-102400"><code>bayes_journal_max_size</code> specifies how large the Bayes journal will grow before it is opportunistically synced. The default value is 102400.</dt>
+<dd>
+
+</dd>
+</dl>
+
+<h1 id="INSTALLATION">INSTALLATION</h1>
+
+<p>The <b>sa-learn</b> command is part of the <b>Mail::SpamAssassin</b> Perl module. Install this as a normal Perl module, using <code>perl -MCPAN -e shell</code>, or by hand.</p>
+
+<h1 id="SEE-ALSO">SEE ALSO</h1>
+
+<p>spamassassin(1) spamc(1) Mail::SpamAssassin(3) Mail::SpamAssassin::ArchiveIterator(3)</p>
+
+<p><http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper</p>
+
+<p><http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and combining algorithms, as used in SpamAssassin</p>
+
+<p><http://web.archive.org/web/20120512230723/http://www.bgl.nu/~glouis/bogofilter/> 'Training on error' page. A discussion of various Bayes training regimes, including 'train on error' and unsupervised training.</p>
+
+<h1 id="PREREQUISITES">PREREQUISITES</h1>
+
+<p><code>Mail::SpamAssassin</code></p>
+
+<h1 id="AUTHORS">AUTHORS</h1>
+
+<p>The SpamAssassin(tm) Project <https://spamassassin.apache.org/></p>
+
+
+</body>
+
+</html>
+
+
Added: spamassassin/site/full/4.0.x/doc/sa-learn.txt
URL: http://svn.apache.org/viewvc/spamassassin/site/full/4.0.x/doc/sa-learn.txt?rev=1906055&view=auto
==============================================================================
--- spamassassin/site/full/4.0.x/doc/sa-learn.txt (added)
+++ spamassassin/site/full/4.0.x/doc/sa-learn.txt Sat Dec 17 11:38:43 2022
@@ -0,0 +1,642 @@
+NAME
+ sa-learn - train SpamAssassin's Bayesian classifier
+
+SYNOPSIS
+ sa-learn [options] [file]...
+
+ sa-learn [options] --dump [ all | data | magic ]
+
+ Options:
+
+ --ham Learn the following messages as ham (non-spam)
+ --spam Learn the following messages as spam
+ --forget Forget the following messages
+ --use-ignores Use bayes_ignore_from and bayes_ignore_to
+ --sync Synchronize the database and the journal if needed
+ --force-expire Force a database sync and expiry run
+ --dbpath <path> Allows commandline override (in bayes_path form)
+ for where to read the Bayes DB from
+ --dump [all|data|magic] Display the contents of the Bayes database
+ Takes optional argument for what to display
+ --regexp <re> For dump only, specifies which tokens to
+ dump based on a regular expression.
+ -f file, --folders=file Read list of files/directories from file
+ --dir Ignored; historical compatibility
+ --file Ignored; historical compatibility
+ --mbox Input sources are in mbox format
+ --mbx Input sources are in mbx format
+ --max-size <b> Skip messages larger than b bytes;
+ defaults to 500 KB, 0 implies no limit
+ --showdots Show progress using dots
+ --progress Show progress using progress bar
+ --no-sync Skip synchronizing the database and journal
+ after learning
+ -L, --local Operate locally, no network accesses. Use
+ of this is recommended, see documentation.
+ --import Migrate data from older version/non DB_File
+ based databases
+ --clear Wipe out existing database
+ --backup Backup, to STDOUT, existing database
+ --restore <filename> Restore a database from filename
+ -u username, --username=username
+ Override username taken from the runtime
+ environment, used with SQL
+ -C path, --configpath=path, --config-file=path
+ Path to standard configuration dir
+ -p prefs, --prefspath=file, --prefs-file=file
+ Set user preferences file
+ --siteconfigpath=path Path for site configs
+ (default: /usr/local/etc/mail/spamassassin)
+ --cf='config line' Additional line of configuration
+ -D, --debug [area,...] Print debugging messages
+ -V, --version Print version
+ -h, --help Print usage message
+
+DESCRIPTION
+ Given a typical selection of your incoming mail classified as spam or
+ ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
+ it to 'learn' what signs are likely to mean spam, and which are likely
+ to mean ham.
+
+ Simply run this command once for each of your mail folders, and it will
+ ''learn'' from the mail therein.
+
+ Note that csh-style *globbing* in the mail folder names is supported; in
+ other words, listing a folder name as "*" will scan every folder that
+ matches. See "Mail::SpamAssassin::ArchiveIterator" for more details.
+
+ If you are using mail boxes in format other than maildir you should use
+ the --mbox or --mbx parameters.
+
+ Files compressed with gzip/bzip2/xz/lz4/lzip/lzo are uncompressed
+ automatically. See "Mail::SpamAssassin::ArchiveIterator" for more
+ details.
+
+ SpamAssassin remembers which mail messages it has learnt already, and
+ will not re-learn those messages again, unless you use the --forget
+ option. Messages learnt as spam will have SpamAssassin markup removed,
+ on the fly.
+
+ If you make a mistake and scan a mail as ham when it is spam, or vice
+ versa, simply rerun this command with the correct classification, and
+ the mistake will be corrected. SpamAssassin will automatically 'forget'
+ the previous indications.
+
+ Users of "spamd" who wish to perform training remotely, over a network,
+ should investigate the "spamc -L" switch.
+
+OPTIONS
+ --ham
+ Learn the input message(s) in the files following the option as ham.
+ If you have previously learnt any of the messages as spam,
+ SpamAssassin will forget them first, then re-learn them as ham.
+ Alternatively, if you have previously learnt them as ham, it'll skip
+ them this time around. If the messages have already been filtered
+ through SpamAssassin, the learner will ignore any modifications
+ SpamAssassin may have made.
+
+ --spam
+ Learn the input message(s) in the files following the option as
+ spam. If you have previously learnt any of the messages as ham,
+ SpamAssassin will forget them first, then re-learn them as spam.
+ Alternatively, if you have previously learnt them as spam, it'll
+ skip them this time around. If the messages have already been
+ filtered through SpamAssassin, the learner will ignore any
+ modifications SpamAssassin may havemmade.
+
+ --folders=*filename*, -f *filename*
+ sa-learn will read in the list of folders from the specified file,
+ one folder per line in the file. If the folder is prefixed with
+ "ham:type:" or "spam:type:", sa-learn will learn that folder
+ appropriately, otherwise the folders will be assumed to be of the
+ type specified by --ham or --spam.
+
+ "type" above is optional, but is the same as the standard for
+ ArchiveIterator: mbox, mbx, dir, file, or detect (the default if not
+ specified).
+
+ --mbox
+ sa-learn will read in the file(s) containing the emails to be
+ learned, and will process them in mbox format (one or more emails
+ per file).
+
+ --mbx
+ sa-learn will read in the file(s) containing the emails to be
+ learned, and will process them in mbx format (one or more emails per
+ file).
+
+ --use-ignores
+ Don't learn the message if a from address matches configuration file
+ item "bayes_ignore_from" or a to address matches "bayes_ignore_to".
+ The option might be used when learning from a large file of messages
+ from which the hammy spam messages or spammy ham messages have not
+ been removed.
+
+ --sync
+ Synchronize the journal and databases. Upon successfully syncing the
+ database with the entries in the journal, the journal file is
+ removed.
+
+ --force-expire
+ Forces an expiry attempt, regardless of whether it may be necessary
+ or not. Note: This doesn't mean any tokens will actually expire.
+ Please see the EXPIRATION section below.
+
+ Note: "--force-expire" also causes the journal data to be
+ synchronized into the Bayes databases.
+
+ --forget
+ Forget the input message(s) in the files following the option as
+ previously learnt.
+
+ --dbpath
+ Allows a commandline override of the *bayes_path* configuration
+ option.
+
+ --dump *option*
+ Display the contents of the Bayes database. Without an option or
+ with the *all* option, all magic tokens and data tokens will be
+ displayed. *magic* will only display magic tokens, and *data* will
+ only display the data tokens.
+
+ Can also use the --regexp *RE* option to specify which tokens to
+ display based on a regular expression.
+
+ --clear
+ Clear an existing Bayes database by removing all traces of the
+ database.
+
+ WARNING: This is destructive and should be used with care.
+
+ --backup
+ Performs a dump of the Bayes database in machine/human readable
+ format.
+
+ The dump will include token and seen data. It is suitable for input
+ back into the --restore command.
+
+ --restore=*filename*
+ Performs a restore of the Bayes database defined by *filename*.
+
+ WARNING: This is a destructive operation, previous Bayes data will
+ be wiped out.
+
+ -h, --help
+ Print help message and exit.
+
+ -u *username*, --username=*username*
+ If specified this username will override the username taken from the
+ runtime environment. You can use this option to specify users in a
+ virtual user configuration when using SQL as the Bayes backend.
+
+ NOTE: This option will not change to the given *username*, it will
+ only attempt to act on behalf of that user. Because of this you will
+ need to have proper permissions to be able to change files owned by
+ *username*. In the case of SQL this generally is not a problem.
+
+ -C *path*, --configpath=*path*, --config-file=*path*
+ Use the specified path for locating the distributed configuration
+ files. Ignore the default directories (usually
+ "/usr/share/spamassassin" or similar).
+
+ --siteconfigpath=*path*
+ Use the specified path for locating site-specific configuration
+ files. Ignore the default directories (usually
+ "/etc/mail/spamassassin" or similar).
+
+ --cf='config line'
+ Add additional lines of configuration directly from the
+ command-line, parsed after the configuration files are read.
+ Multiple --cf arguments can be used, and each will be considered a
+ separate line of configuration.
+
+ -p *prefs*, --prefspath=*prefs*, --prefs-file=*prefs*
+ Read user score preferences from *prefs* (usually
+ "$HOME/.spamassassin/user_prefs").
+
+ --progress
+ Prints a progress bar (to STDERR) showing the current progress. In
+ the case where no valid terminal is found this option will behave
+ very much like the --showdots option.
+
+ -D [*area,...*], --debug [*area,...*]
+ Produce debugging output. If no areas are listed, all debugging
+ information is printed. Diagnostic output can also be enabled for
+ each area individually; *area* is the area of the code to
+ instrument. For example, to produce diagnostic output on bayes,
+ learn, and dns, use:
+
+ spamassassin -D bayes,learn,dns
+
+ Use an empty string (-D '') to indicate no areas when the next item
+ on the command line is a path, to prevent the path from being parsed
+ as an area.
+
+ For more information about which areas (also known as channels) are
+ available, please see the documentation at:
+
+ C<https://wiki.apache.org/spamassassin/DebugChannels>
+
+ Higher priority informational messages that are suitable for logging
+ in normal circumstances are available with an area of "info".
+
+ --no-sync
+ Skip the slow synchronization step which normally takes place after
+ changing database entries. If you plan to learn from many folders in
+ a batch, or to learn many individual messages one-by-one, it is
+ faster to use this switch and run "sa-learn --sync" once all the
+ folders have been scanned.
+
+ Clarification: The state of *--no-sync* overrides the
+ *bayes_learn_to_journal* configuration option. If not specified,
+ sa-learn will learn to the database directly. If specified, sa-learn
+ will learn to the journal file.
+
+ Note: *--sync* and *--no-sync* can be specified on the same
+ commandline, which is slightly confusing. In this case, the
+ *--no-sync* option is ignored since there is no learn operation.
+
+ -L, --local
+ Do not perform any network accesses while learning details about the
+ mail messages. This should be normally used, as there really isn't
+ anything Bayes can learn from network lookup results. Official
+ SpamAssassin plugins do not currently do any network lookups when
+ learning, but it's possible that third party ones might.
+
+ --import
+ If you previously used SpamAssassin's Bayesian learner without the
+ "DB_File" module installed, it will have created files in other
+ formats, such as "GDBM_File", "NDBM_File", or "SDBM_File". This
+ switch allows you to migrate that old data into the "DB_File"
+ format. It will overwrite any data currently in the "DB_File".
+
+ Can also be used with the --dbpath *path* option to specify the
+ location of the Bayes files to use.
+
+MIGRATION
+ There are now multiple backend storage modules available for storing
+ user's bayesian data. As such you might want to migrate from one backend
+ to another. Here is a simple procedure for migrating from one backend to
+ another.
+
+ Note that if you have individual user databases you will have to perform
+ a similar procedure for each one of them.
+
+ sa-learn --sync
+ This will sync any outstanding journal entries
+
+ sa-learn --backup > backup.txt
+ This will save all your Bayes data to a plain text file.
+
+ sa-learn --clear
+ This is optional, but good to do to clear out the old database.
+
+ Repeat!
+ At this point, if you have multiple databases, you should perform
+ the procedure above for each of them. (i.e. each user's database
+ needs to be backed up before continuing.)
+
+ Switch backends
+ Once you have backed up all databases you can update your
+ configuration for the new database backend. This will involve at
+ least the bayes_store_module config option and may involve some
+ additional config options depending on what is required by the
+ module. (For example, you may need to configure an SQL database.)
+
+ sa-learn --restore backup.txt
+ Again, you need to do this for every database.
+
+ If you are migrating to SQL you can make use of the -u <username> option
+ in sa-learn to populate each user's database. Otherwise, you must run
+ sa-learn as the user who database you are restoring.
+
+INTRODUCTION TO BAYESIAN FILTERING
+ (Thanks to Michael Bell for this section!)
+
+ For a more lengthy description of how this works, go to
+ http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably
+ readable, even if statistics make me break out in hives.
+
+ The short semi-inaccurate version: Given training, a spam heuristics
+ engine can take the most "spammy" and "hammy" words and apply
+ probabilistic analysis. Furthermore, once given a basis for the
+ analysis, the engine can continue to learn iteratively by applying both
+ the non-Bayesian and Bayesian rulesets together to create evolving
+ "intelligence".
+
+ SpamAssassin 2.50 and later supports Bayesian spam analysis, in the form
+ of the BAYES rules. This is a new feature, quite powerful, and is
+ disabled until enough messages have been learnt.
+
+ The pros of Bayesian spam analysis:
+
+ Can greatly reduce false positives and false negatives.
+ It learns from your mail, so it is tailored to your unique e-mail
+ flow.
+
+ Once it starts learning, it can continue to learn from SpamAssassin and
+ improve over time.
+
+ And the cons:
+
+ A decent number of messages are required before results are useful for
+ ham/spam determination.
+ It's hard to explain why a message is or isn't marked as spam.
+ i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy to
+ understand. If it generates a false positive or false negative, it
+ is fairly easy to understand why.
+
+ With Bayesian analysis, it's all probabilities - "because the past
+ says it is likely as this falls into a probabilistic distribution
+ common to past spam in your systems". Tell that to your users! Tell
+ that to the client when he asks "what can I do to change this". (By
+ the way, the answer in this case is "use welcomelisting".)
+
+ It will take disk space and memory.
+ The databases it maintains take quite a lot of resources to store
+ and use.
+
+GETTING STARTED
+ Still interested? Ok, here's the guidelines for getting this working.
+
+ First a high-level overview:
+
+ Build a significant sample of both ham and spam.
+ I suggest several thousand of each, placed in SPAM and HAM
+ directories or mailboxes. Yes, you MUST hand-sort this - otherwise
+ the results won't be much better than SpamAssassin on its own.
+ Verify the spamminess/haminess of EVERY message. You're urged to
+ avoid using a publicly available corpus (sample) - this must be
+ taken from YOUR mail server, if it is to be statistically useful.
+ Otherwise, the results may be pretty skewed.
+
+ Use this tool to teach SpamAssassin about these samples, like so:
+ sa-learn --spam /path/to/spam/folder
+ sa-learn --ham /path/to/ham/folder
+ sa-learn --ham hampath1 hampath2 --spam spampath1 spampath2
+ ...
+
+ Let SpamAssassin proceed, learning stuff. When it finds ham and spam
+ it will add the "interesting tokens" to the database.
+
+ If you need SpamAssassin to forget about specific messages, use the
+ --forget option.
+ This can be applied to either ham or spam that has run through the
+ sa-learn processes. It's a bit of a hammer, really, lowering the
+ weighting of the specific tokens in that message (only if that
+ message has been processed before).
+
+ Learning from single messages uses a command like this:
+ sa-learn --ham --no-sync mailmessage
+
+ This is handy for binding to a key in your mail user agent. It's
+ very fast, as all the time-consuming stuff is deferred until you run
+ with the "--sync" option.
+
+ Autolearning is enabled by default
+ If you don't have a corpus of mail saved to learn, you can let
+ SpamAssassin automatically learn the mail that you receive. If you
+ are autolearning from scratch, the amount of mail you receive will
+ determine how long until the BAYES_* rules are activated.
+
+EFFECTIVE TRAINING
+ Learning filters require training to be effective. If you don't train
+ them, they won't work. In addition, you need to train them with new
+ messages regularly to keep them up-to-date, or their data will become
+ stale and impact accuracy.
+
+ You need to train with both spam *and* ham mails. One type of mail alone
+ will not have any effect.
+
+ Note that if your mail folders contain things like forwarded spam,
+ discussions of spam-catching rules, etc., this will cause trouble. You
+ should avoid scanning those messages if possible. (An easy way to do
+ this is to move them aside, into a folder which is not scanned.)
+
+ If the messages you are learning from have already been filtered through
+ SpamAssassin, the learner will compensate for this. In effect, it learns
+ what each message would look like if you had run "spamassassin -d" over
+ it in advance.
+
+ Another thing to be aware of, is that typically you should aim to train
+ with at least 1000 messages of spam, and 1000 ham messages, if possible.
+ More is better, but anything over about 5000 messages does not improve
+ accuracy significantly in our tests.
+
+ Be careful that you train from the same source -- for example, if you
+ train on old spam, but new ham mail, then the classifier will think that
+ a mail with an old date stamp is likely to be spam.
+
+ It's also worth noting that training with a very small quantity of ham,
+ will produce atrocious results. You should aim to train with at least
+ the same amount (or more if possible!) of ham data than spam.
+
+ On an on-going basis, it is best to keep training the filter to make
+ sure it has fresh data to work from. There are various ways to do this:
+
+ 1. Supervised learning
+ This means keeping a copy of all or most of your mail, separated
+ into spam and ham piles, and periodically re-training using those.
+ It produces the best results, but requires more work from you, the
+ user.
+
+ (An easy way to do this, by the way, is to create a new folder for
+ 'deleted' messages, and instead of deleting them from other folders,
+ simply move them in there instead. Then keep all spam in a separate
+ folder and never delete it. As long as you remember to move
+ misclassified mails into the correct folder set, it is easy enough
+ to keep up to date.)
+
+ 2. Unsupervised learning from Bayesian classification
+ Another way to train is to chain the results of the Bayesian
+ classifier back into the training, so it reinforces its own
+ decisions. This is only safe if you then retrain it based on any
+ errors you discover.
+
+ SpamAssassin does not support this method, due to experimental
+ results which strongly indicate that it does not work well, and
+ since Bayes is only one part of the resulting score presented to the
+ user (while Bayes may have made the wrong decision about a mail, it
+ may have been overridden by another system).
+
+ 3. Unsupervised learning from SpamAssassin rules
+ Also called 'auto-learning' in SpamAssassin. Based on statistical
+ analysis of the SpamAssassin success rates, we can automatically
+ train the Bayesian database with a certain degree of confidence that
+ our training data is accurate.
+
+ It should be supplemented with some supervised training in addition,
+ if possible.
+
+ This is the default, but can be turned off by setting the
+ SpamAssassin configuration parameter "bayes_auto_learn" to 0.
+
+ 4. Mistake-based training
+ This means training on a small number of mails, then only training
+ on messages that SpamAssassin classifies incorrectly. This works,
+ but it takes longer to get it right than a full training session
+ would.
+
+FILES
+ sa-learn and the other parts of SpamAssassin's Bayesian learner, use a
+ set of persistent database files to store the learnt tokens, as follows.
+
+ bayes_toks
+ The database of tokens, containing the tokens learnt, their count of
+ occurrences in ham and spam, and the timestamp when the token was
+ last seen in a message.
+
+ This database also contains some 'magic' tokens, as follows: the
+ version number of the database, the number of ham and spam messages
+ learnt, the number of tokens in the database, and timestamps of: the
+ last journal sync, the last expiry run, the last expiry token
+ reduction count, the last expiry timestamp delta, the oldest token
+ timestamp in the database, and the newest token timestamp in the
+ database.
+
+ This is a database file, using "DB_File". The database 'version
+ number' is 0 for databases from 2.5x, 1 for databases from certain
+ 2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later
+ releases.
+
+ bayes_seen
+ A map of Message-Id and some data from headers and body to what that
+ message was learnt as. This is used so that SpamAssassin can avoid
+ re-learning a message it has already seen, and so it can reverse the
+ training if you later decide that message was learnt incorrectly.
+
+ This is a database file, using "DB_File".
+
+ bayes_journal
+ While SpamAssassin is scanning mails, it needs to track which tokens
+ it uses in its calculations. To avoid the contention of having each
+ SpamAssassin process attempting to gain write access to the Bayes
+ DB, the token timestamps are written to a 'journal' file which will
+ later (either automatically or via "sa-learn --sync") be used to
+ synchronize the Bayes DB.
+
+ Also, through the use of "bayes_learn_to_journal", or when using the
+ "--no-sync" option with sa-learn, the actual learning data will take
+ be placed into the journal for later synchronization. This is
+ typically useful for high-traffic sites to avoid the same contention
+ as stated above.
+
+EXPIRATION
+ Since SpamAssassin can auto-learn messages, the Bayes database files
+ could increase perpetually until they fill your disk. To control this,
+ SpamAssassin performs journal synchronization and bayes expiration
+ periodically when certain criteria (listed below) are met.
+
+ SpamAssassin can sync the journal and expire the DB tokens either
+ manually or opportunistically. A journal sync is due if *--sync* is
+ passed to sa-learn (manual), or if the following is true
+ (opportunistic):
+
+ - bayes_journal_max_size does not equal 0 (means don't sync)
+ - the journal file exists
+
+ and either:
+
+ - the journal file has a size greater than bayes_journal_max_size
+
+ or
+
+ - a journal sync has previously occurred, and at least 1 day has passed
+ since that sync
+
+ Expiry is due if *--force-expire* is passed to sa-learn (manual), or if
+ all of the following are true (opportunistic):
+
+ - the last expire was attempted at least 12hrs ago
+ - bayes_auto_expire does not equal 0
+ - the number of tokens in the DB is > 100,000
+ - the number of tokens in the DB is > bayes_expiry_max_db_size
+ - there is at least a 12 hr difference between the oldest and newest
+ token atimes
+
+ EXPIRE LOGIC
+ If either the manual or opportunistic method causes an expire run to
+ start, here is the logic that is used:
+
+ - figure out how many tokens to keep. take the larger of either
+ bayes_expiry_max_db_size * 75% or 100,000 tokens. therefore, the goal
+ reduction is number of tokens - number of tokens to keep.
+ - if the reduction number is < 1000 tokens, abort (not worth the
+ effort).
+ - if an expire has been done before, guesstimate the new atime delta
+ based on the old atime delta. (new_atime_delta = old_atime_delta *
+ old_reduction_count / goal)
+ - if no expire has been done before, or the last expire looks "weird",
+ do an estimation pass. The definition of "weird" is:
+
+ - last expire over 30 days ago
+ - last atime delta was < 12 hrs
+ - last reduction count was < 1000 tokens
+ - estimated new atime delta is < 12 hrs
+ - the difference between the last reduction count and the goal
+ reduction count is > 50%
+
+ ESTIMATION PASS LOGIC
+ Go through each of the DB's tokens. Starting at 12hrs, calculate whether
+ or not the token would be expired (based on the difference between the
+ token's atime and the db's newest token atime) and keep the count. Work
+ out from 12hrs exponentially by powers of 2. ie: 12hrs * 1, 12hrs * 2,
+ 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512 (6144hrs, or 256
+ days).
+
+ The larger the delta, the smaller the number of tokens that will be
+ expired. Conversely, the number of tokens goes up as the delta gets
+ smaller. So starting at the largest atime delta, figure out which delta
+ will expire the most tokens without going above the goal expiration
+ count. Use this to choose the atime delta to use, unless one of the
+ following occurs:
+
+ - the largest atime (smallest reduction count) would expire too many
+ tokens. this means the learned tokens are mostly old and there needs to
+ be new tokens learned before an expire can occur.
+ - all of the atime choices result in 0 tokens being removed. this means
+ the tokens are all newer than 12 hours and there needs to be new tokens
+ learned before an expire can occur.
+ - the number of tokens that would be removed is < 1000. the benefit
+ isn't worth the effort. more tokens need to be learned.
+
+ If the expire run gets past this point, it will continue to the end. A
+ new DB is created since the majority of DB libraries don't shrink the DB
+ file when tokens are removed. So we do the "create new, migrate old to
+ new, remove old, rename new" shuffle.
+
+ EXPIRY RELATED CONFIGURATION SETTINGS
+ "bayes_auto_expire" is used to specify whether or not SpamAssassin ought
+ to opportunistically attempt to expire the Bayes database. The default
+ is 1 (yes).
+ "bayes_expiry_max_db_size" specifies both the auto-expire token count
+ point, as well as the resulting number of tokens after expiry as
+ described above. The default value is 150,000, which is roughly
+ equivalent to a 6Mb database file if you're using DB_File.
+ "bayes_journal_max_size" specifies how large the Bayes journal will grow
+ before it is opportunistically synced. The default value is 102400.
+
+INSTALLATION
+ The sa-learn command is part of the Mail::SpamAssassin Perl module.
+ Install this as a normal Perl module, using "perl -MCPAN -e shell", or
+ by hand.
+
+SEE ALSO
+ spamassassin(1) spamc(1) Mail::SpamAssassin(3)
+ Mail::SpamAssassin::ArchiveIterator(3)
+
+ <http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper
+
+ <http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and
+ combining algorithms, as used in SpamAssassin
+
+ <http://web.archive.org/web/20120512230723/http://www.bgl.nu/~glouis/bog
+ ofilter/> 'Training on error' page. A discussion of various Bayes
+ training regimes, including 'train on error' and unsupervised training.
+
+PREREQUISITES
+ "Mail::SpamAssassin"
+
+AUTHORS
+ The SpamAssassin(tm) Project <https://spamassassin.apache.org/>
+
Added: spamassassin/site/full/4.0.x/doc/sa-update.html
URL: http://svn.apache.org/viewvc/spamassassin/site/full/4.0.x/doc/sa-update.html?rev=1906055&view=auto
==============================================================================
--- spamassassin/site/full/4.0.x/doc/sa-update.html (added)
+++ spamassassin/site/full/4.0.x/doc/sa-update.html Sat Dec 17 11:38:43 2022
@@ -0,0 +1,279 @@
+<?xml version="1.0" ?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+<title>sa-update - automate SpamAssassin rule updates</title>
+<meta http-equiv="content-type" content="text/html; charset=utf-8" />
+<link rev="made" href="mailto:root@localhost" />
+</head>
+
+<body>
+
+
+
+<ul id="index">
+ <li><a href="#NAME">NAME</a></li>
+ <li><a href="#SYNOPSIS">SYNOPSIS</a></li>
+ <li><a href="#DESCRIPTION">DESCRIPTION</a></li>
+ <li><a href="#OPTIONS">OPTIONS</a></li>
+ <li><a href="#EXIT-CODES">EXIT CODES</a></li>
+ <li><a href="#SEE-ALSO">SEE ALSO</a></li>
+ <li><a href="#PREREQUISITES">PREREQUISITES</a></li>
+ <li><a href="#BUGS">BUGS</a></li>
+ <li><a href="#AUTHORS">AUTHORS</a></li>
+ <li><a href="#LICENSE-AND-COPYRIGHT">LICENSE AND COPYRIGHT</a></li>
+</ul>
+
+<h1 id="NAME">NAME</h1>
+
+<p>sa-update - automate SpamAssassin rule updates</p>
+
+<h1 id="SYNOPSIS">SYNOPSIS</h1>
+
+<p><b>sa-update</b> [options]</p>
+
+<p>Options:</p>
+
+<pre><code>--channel channel Retrieve updates from this channel
+ Use multiple times for multiple channels
+--channelfile file Retrieve updates from the channels in the file
+--checkonly Check for update availability, do not install
+--install file Install updates directly from this file. Signature
+ verification will use "file.asc", or "file.sha512"
+ or "file.sha256".
+--allowplugins Allow updates to load plugin code (DANGEROUS)
+--gpgkey key Trust the key id to sign releases
+ Use multiple times for multiple keys
+--gpgkeyfile file Trust the key ids in the file to sign releases
+--gpghomedir path Store the GPG keyring in this directory
+--gpg and --nogpg Use (or do not use) GPG to verify updates
+ (--gpg is assumed by use of the above
+ --gpgkey and --gpgkeyfile options)
+--import file Import GPG key(s) from file into sa-update's
+ keyring. Use multiple times for multiple files
+--updatedir path Directory to place updates, defaults to the
+ SpamAssassin site rules directory
+ (default: /var/lib/spamassassin/4.000000)
+--refreshmirrors Force the MIRRORED.BY file to be updated
+--forcemirror url Use a specific mirror instead of downloading from
+ official mirrors
+--httputil util Force used download tool. By default first found
+ from these is used: curl, wget, fetch, lwp
+--score-multiplier x.x Adjust all scores from update channel, multiply
+ with given value (integer or float).
+--score-limit x.x Adjust all scores from update channel, limit
+ to given value (integer or float). Limiting
+ is done after possible multiply operation.
+-D, --debug [area=n,...] Print debugging messages
+-v, --verbose Be verbose, like print updated channel names;
+ For more verbosity specify multiple times
+-V, --version Print version
+-h, --help Print usage message
+-4 Force using the inet protocol (IPv4), not inet6
+-6 Force using the inet6 protocol (IPv6), not inet</code></pre>
+
+<h1 id="DESCRIPTION">DESCRIPTION</h1>
+
+<p>sa-update automates the process of downloading and installing new rules and configuration, based on channels. The default channel is <i>updates.spamassassin.org</i>, which has updated rules since the previous release.</p>
+
+<p>Update archives are verified using GPG signatures by default. If GPG is disabled (not recommended), file integrity is checked with SHA512 or SHA256 checksums.</p>
+
+<p>Note that <code>sa-update</code> will not restart <code>spamd</code> or otherwise cause a scanner to reload the now-updated ruleset automatically. Instead, <code>sa-update</code> is typically used in something like the following manner:</p>
+
+<pre><code>sa-update && /etc/init.d/spamassassin reload</code></pre>
+
+<p>This works because <code>sa-update</code> only returns an exit status of <code>0</code> if it has successfully downloaded and installed an updated ruleset.</p>
+
+<p>The program sa-update uses the underlying operating system umask for the updated rule files it installs. You may wish to run sa-update from a script that sets the umask prior to calling sa-update. For example:</p>
+
+<pre><code>#!/bin/sh
+umask 022
+sa-update</code></pre>
+
+<h1 id="OPTIONS">OPTIONS</h1>
+
+<dl>
+
+<dt id="channel"><b>--channel</b></dt>
+<dd>
+
+<p>sa-update can update multiple channels at the same time. By default, it will only access "updates.spamassassin.org", but more channels can be specified via this option. If there are multiple additional channels, use the option multiple times, once per channel. i.e.:</p>
+
+<pre><code>sa-update --channel foo.example.com --channel bar.example.com</code></pre>
+
+</dd>
+<dt id="channelfile"><b>--channelfile</b></dt>
+<dd>
+
+<p>Similar to the <b>--channel</b> option, except specify the additional channels in a file instead of on the commandline. This is useful when there are a lot of additional channels.</p>
+
+</dd>
+<dt id="checkonly"><b>--checkonly</b></dt>
+<dd>
+
+<p>Only check if an update is available, don't actually download and install it. The exit code will be <code>0</code> or <code>1</code> as described below.</p>
+
+</dd>
+<dt id="install"><b>--install</b></dt>
+<dd>
+
+<p>Install updates "offline", from the named tar.gz file, instead of performing DNS lookups and HTTP invocations.</p>
+
+<p>Files named <b>file</b>.asc, <b>file</b>.sha512, or <b>file</b>.sha256 will be used for GPG signature, and the SHA256 and SHA512 checksums, respectively. The filename provided must contain a version number of at least 3 digits, which will be used as the channel's update version number.</p>
+
+<p>Multiple <b>--channel</b> switches cannot be used with <b>--install</b>. To install multiple channels from tarballs, run <code>sa-update</code> multiple times with different <b>--channel</b> and <b>--install</b> switches, e.g.:</p>
+
+<pre><code>sa-update --channel foo.example.com --install foo-34958.tgz
+sa-update --channel bar.example.com --install bar-938455.tgz</code></pre>
+
+</dd>
+<dt id="allowplugins"><b>--allowplugins</b></dt>
+<dd>
+
+<p>Allow downloaded updates to activate plugins. The default is not to activate plugins; any <code>loadplugin</code> or <code>tryplugin</code> lines will be commented in the downloaded update rules files.</p>
+
+<p>You should never enable this for 3rd party update channels, since plugins can execute unrestricted code on your system, even possibly as root! This includes spamassassin official updates, which have no need to include running code.</p>
+
+<p>Use --reallyallowplugins option to bypass warnings and make it work.</p>
+
+</dd>
+<dt id="gpg---nogpg"><b>--gpg</b>, <b>--nogpg</b></dt>
+<dd>
+
+<p>sa-update by default will verify update archives by use of GPG signature.</p>
+
+<p>If you wish to skip GPG verification (very unsafe), you can use the <b>--nogpg</b> option to disable its use. Use of the following gpgkey-related options will override <b>--nogpg</b> and keep GPG verification enabled.</p>
+
+<p>If GPG is disabled, only SHA512 or SHA256 checksums are used to verify whether or not the downloaded archive has been corrupted, but it does not offer any form of security regarding whether or not the downloaded archive is legitimate (aka: non-modifed by evildoers).</p>
+
+<p>Note: Only GnuPG is supported (ie: not any other PGP software).</p>
+
+</dd>
+<dt id="gpgkey"><b>--gpgkey</b></dt>
+<dd>
+
+<p>sa-update has the concept of "release trusted" GPG keys. When an archive is downloaded and the signature verified, sa-update requires that the signature be from one of these "release trusted" keys or else verification fails. This prevents third parties from manipulating the files on a mirror, for instance, and signing with their own key.</p>
+
+<p>By default, sa-update trusts key ids <code>24F434CE</code> and <code>5244EC45</code>, which are the standard SpamAssassin release key and its sub-key. Use this option to trust additional keys. See the <b>--import</b> option for how to add keys to sa-update's keyring. For sa-update to use a key it must be in sa-update's keyring and trusted.</p>
+
+<p>For multiple keys, use the option multiple times. i.e.:</p>
+
+<pre><code>sa-update --gpgkey E580B363 --gpgkey 298BC7D0</code></pre>
+
+<p>Note: use of this option automatically enables GPG verification.</p>
+
+</dd>
+<dt id="gpgkeyfile"><b>--gpgkeyfile</b></dt>
+<dd>
+
+<p>Similar to the <b>--gpgkey</b> option, except specify the additional keys in a file instead of on the commandline. This is extremely useful when there are a lot of additional keys that you wish to trust.</p>
+
+</dd>
+<dt id="gpghomedir"><b>--gpghomedir</b></dt>
+<dd>
+
+<p>Specify a directory path to use as a storage area for the <code>sa-update</code> GPG keyring. By default, this is</p>
+
+<pre><code>/etc/mail/spamassassin/sa-update-keys</code></pre>
+
+</dd>
+<dt id="import"><b>--import</b></dt>
+<dd>
+
+<p>Use to import GPG key(s) from a file into the sa-update keyring which is located in the directory specified by <b>--gpghomedir</b>. Before using channels from third party sources, you should use this option to import the GPG key(s) used by those channels. You must still use the <b>--gpgkey</b> or <b>--gpgkeyfile</b> options above to get sa-update to trust imported keys.</p>
+
+<p>To import multiple keys, use the option multiple times. i.e.:</p>
+
+<pre><code>sa-update --import channel1-GPG.KEY --import channel2-GPG.KEY</code></pre>
+
+<p>Note: use of this option automatically enables GPG verification.</p>
+
+</dd>
+<dt id="refreshmirrors"><b>--refreshmirrors</b></dt>
+<dd>
+
+<p>Force the list of sa-update mirrors for each channel, stored in the MIRRORED.BY file, to be updated. By default, the MIRRORED.BY file will be cached for up to 7 days after each time it is downloaded.</p>
+
+</dd>
+<dt id="forcemirror"><b>--forcemirror</b></dt>
+<dd>
+
+<p>Force the download from a specific host instead of relying on mirrors listed in MIRRORED.BY.</p>
+
+</dd>
+<dt id="updatedir"><b>--updatedir</b></dt>
+<dd>
+
+<p>By default, <code>sa-update</code> will use the system-wide rules update directory:</p>
+
+<pre><code>/var/lib/spamassassin/4.000000</code></pre>
+
+<p>If the updates should be stored in another location, specify it here.</p>
+
+<p>Note that use of this option is not recommended; if you're just using sa-update to download updated rulesets for a scanner, and sa-update is placing updates in the wrong directory, you probably need to rebuild SpamAssassin with different <code>Makefile.PL</code> arguments, instead of overriding sa-update's runtime behaviour.</p>
+
+</dd>
+<dt id="D-area-...---debug-area"><b>-D</b> [<i>area,...</i>], <b>--debug</b> [<i>area,...</i>]</dt>
+<dd>
+
+<p>Produce debugging output. If no areas are listed, all debugging information is printed. Diagnostic output can also be enabled for each area individually; <i>area</i> is the area of the code to instrument. For example, to produce diagnostic output on channel, gpg, and http, use:</p>
+
+<pre><code>sa-update -D channel,gpg,http</code></pre>
+
+<p>For more information about which areas (also known as channels) are available, please see the documentation at <a href="https://wiki.apache.org/spamassassin/DebugChannels">https://wiki.apache.org/spamassassin/DebugChannels</a>.</p>
+
+</dd>
+<dt id="h---help"><b>-h</b>, <b>--help</b></dt>
+<dd>
+
+<p>Print help message and exit.</p>
+
+</dd>
+<dt id="V---version"><b>-V</b>, <b>--version</b></dt>
+<dd>
+
+<p>Print sa-update version and exit.</p>
+
+</dd>
+</dl>
+
+<h1 id="EXIT-CODES">EXIT CODES</h1>
+
+<p>In absence of a --checkonly option, an exit code of <code>0</code> means: an update was available, and was downloaded and installed successfully. If --checkonly was specified, an exit code of <code>0</code> means: an update was available.</p>
+
+<p>An exit code of <code>1</code> means no fresh updates were available.</p>
+
+<p>An exit code of <code>2</code> means that at least one update is available but that a lint check of the site pre files failed. The site pre files must pass a lint check before any updates are attempted.</p>
+
+<p>An exit code of <code>3</code> means that at least one update succeeded while other channels failed. If using sa-compile, you should proceed with it.</p>
+
+<p>An exit code of <code>4</code> or higher, indicates that errors occurred while attempting to download and extract updates, and no channels were updated.</p>
+
+<h1 id="SEE-ALSO">SEE ALSO</h1>
+
+<p>Mail::SpamAssassin(3) Mail::SpamAssassin::Conf(3) spamassassin(1) spamd(1) <https://wiki.apache.org/spamassassin/RuleUpdates></p>
+
+<h1 id="PREREQUISITES">PREREQUISITES</h1>
+
+<p><code>Mail::SpamAssassin</code></p>
+
+<h1 id="BUGS">BUGS</h1>
+
+<p>See <https://issues.apache.org/SpamAssassin/></p>
+
+<h1 id="AUTHORS">AUTHORS</h1>
+
+<p>The Apache SpamAssassin(tm) Project <https://spamassassin.apache.org/></p>
+
+<h1 id="LICENSE-AND-COPYRIGHT">LICENSE AND COPYRIGHT</h1>
+
+<p>SpamAssassin is distributed under the Apache License, Version 2.0, as described in the file <code>LICENSE</code> included with the distribution.</p>
+
+<p>Copyright (C) 2015 The Apache Software Foundation</p>
+
+
+</body>
+
+</html>
+
+