Positive / Negative (was: Re: spamc -L apparently not working properly)

[Karsten Bräckelmann <gu...@rudersport.de>]

On Sun, 2013-11-10 at 03:32 -0200, Sergio Durigan Junior wrote:
> On Sunday, November 10 2013, Karsten Bräckelmann wrote:

> For all messages that I received since I started using SA (about 20
> messages, of which 5 were false-negatives, and the rest were
> true-negatives), [...]

Given you state below no spam has been identified yet, you're confusing
terms.

SA tests for spam. Thus a positive result is "classified spam", and "not
spam" is a negative test result. True means the result is correct,
whereas false indicates a mis-classification by the test.

False (mis-classified) negatives (rated not-spam) are spam, which SA
failed to classify spam.


If you prefer, refer to them as missed spam, or (in)correctly classified
ham and spam.


> I do receive spam.  About 1 or 2 per day.  But so far SA hasn't been
> able to catch any of them, and all spam I receive has been marked as ham
> so far.  The message headers are OK, there is nothing apparently wrong
> with SA, but it is just not catching most of my spam.  I assume this is
> normal behavior since I just started using SA a few days ago.

No, that is not normal. In fact, since no spam has been identified at
all yet, there is something really broken or mis-configured.

I suggest to start a new thread (no reply) about this. For starters,
we'd need details about your environment and how you set up SA. Plus
some X-Spam-Status headers of ham and (missed) spam.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Positive / Negative

[Sergio Durigan Junior <se...@sergiodj.net>]

On Monday, November 11 2013, Karsten Bräckelmann wrote:

> 'sa-learn --dump magic' still shows less than 200 nham / nspam, right?

Yes, it does.

> Until that issue is resolved, please keep the spam for potential further
> post-receiving tests.

Will certainly do.

> Not strictly SA configuration, but you probably want to change the
> following Debian defaults in /etc/default/spamassassin
>
>   ENABLED=0
>   CRON=0
>
> and enable the spamd daemon system-wide, as well as sa-update.
>
> If you didn't yet run sa-update, do so now. Restart spamd afterward.
> FWIW, this counts as "modifying SA config", since it updates the stock
> rule-set.

Oh, I did that, yeah.  I meant to say that I did not touch in any file
under /etc/spamassassin.  So my /etc/spamassassin/local.cf, for example,
is exactly what is shipped with Debian.

Thanks,

-- 
Sergio

Re: Positive / Negative

[Karsten Bräckelmann <gu...@rudersport.de>]

On Mon, 2013-11-11 at 00:34 -0200, Sergio Durigan Junior wrote:
> On Sunday, November 10 2013, Karsten Bräckelmann wrote:
> > Given you state below no spam has been identified yet, you're confusing
> > terms.

Gnah. I was falsely thinking "received" when I wrote "identified" there.

> I don't think I am confusing terms.

True, my bad, sorry.

> > If you prefer, refer to them as missed spam, or (in)correctly classified
> > ham and spam.
> 
> OK, I will make use of those terms if it makes things clearer for you.

That however should not be possible. I guess I am entirely capable of
handling the terms FP and FN... ;)


> Indeed, no spam has been classified at all since I started running SA.
> 
> An interesting fact is that, before I started using SA, I had some spams
> left in my INBOX.  Well, when I decided that it was time to use SA, I
> manually fed those spams to spamc (for testing purposes), and SA
> correctly identified almost all of them!  But now, as I said, SA is
> failing to classify the spam I've been receiving.

'sa-learn --dump magic' still shows less than 200 nham / nspam, right?

> > I suggest to start a new thread (no reply) about this. For starters,
> > we'd need details about your environment and how you set up SA. Plus
> > some X-Spam-Status headers of ham and (missed) spam.
> 
> OK, fair enough.  Unfortunately, I don't have any spam messages left.  I
> used them all to feed sa-learn, and then deleted them.  But as soon as I
> get another misclassified spam, I will start another thread on this
> topic, with all the information requested

Until that issue is resolved, please keep the spam for potential further
post-receiving tests.


> (BTW, I am using a default Debian SA configuration, and did not modify
> anything so far).

Not strictly SA configuration, but you probably want to change the
following Debian defaults in /etc/default/spamassassin

  ENABLED=0
  CRON=0

and enable the spamd daemon system-wide, as well as sa-update.

If you didn't yet run sa-update, do so now. Restart spamd afterward.
FWIW, this counts as "modifying SA config", since it updates the stock
rule-set.


-- 
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Positive / Negative

[Sergio Durigan Junior <se...@sergiodj.net>]

On Sunday, November 10 2013, Karsten Bräckelmann wrote:

> On Sun, 2013-11-10 at 03:32 -0200, Sergio Durigan Junior wrote:
>> On Sunday, November 10 2013, Karsten Bräckelmann wrote:
>
>> For all messages that I received since I started using SA (about 20
>> messages, of which 5 were false-negatives, and the rest were
>> true-negatives), [...]
>
> Given you state below no spam has been identified yet, you're confusing
> terms.
>
> SA tests for spam. Thus a positive result is "classified spam", and "not
> spam" is a negative test result. True means the result is correct,
> whereas false indicates a mis-classification by the test.
>
> False (mis-classified) negatives (rated not-spam) are spam, which SA
> failed to classify spam.

I don't think I am confusing terms.

false-negative: spam that got classified as ham
false-positive: ham that got classified as spam
true-negative: ham
true-positive: spam

Maybe my terms aren't the correct ones, and if that's the case, sorry
about it.

> If you prefer, refer to them as missed spam, or (in)correctly classified
> ham and spam.

OK, I will make use of those terms if it makes things clearer for you.

>> I do receive spam.  About 1 or 2 per day.  But so far SA hasn't been
>> able to catch any of them, and all spam I receive has been marked as ham
>> so far.  The message headers are OK, there is nothing apparently wrong
>> with SA, but it is just not catching most of my spam.  I assume this is
>> normal behavior since I just started using SA a few days ago.
>
> No, that is not normal. In fact, since no spam has been identified at
> all yet, there is something really broken or mis-configured.

Indeed, no spam has been classified at all since I started running SA.

An interesting fact is that, before I started using SA, I had some spams
left in my INBOX.  Well, when I decided that it was time to use SA, I
manually fed those spams to spamc (for testing purposes), and SA
correctly identified almost all of them!  But now, as I said, SA is
failing to classify the spam I've been receiving.

> I suggest to start a new thread (no reply) about this. For starters,
> we'd need details about your environment and how you set up SA. Plus
> some X-Spam-Status headers of ham and (missed) spam.

OK, fair enough.  Unfortunately, I don't have any spam messages left.  I
used them all to feed sa-learn, and then deleted them.  But as soon as I
get another misclassified spam, I will start another thread on this
topic, with all the information requested (BTW, I am using a default
Debian SA configuration, and did not modify anything so far).

Thanks,

-- 
Sergio