Re: Stanford logistic regression results

[Justin Mason <jm...@jmason.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


BTW did anything ever come of this?  also what do the plots look
like?  (I'm sure I saw them at the time, but it'd be nice to
get them on a wiki.)

PS: the ham misclassification constant factor "alpha" maps to TCR's
"lambda" I think.

- --j.

Daniel Quinlan writes:
> I'm not sure if I forwarded these before, but as part of being here at
> CEAS in Stanford, we've been talking a bit to Mike Brzozowski and Andrew
> Y. Ng about these results from last year on 3.0.
> 
> I think the (very preliminary) logistic regression results warrant
> further testing.
> 
> Mike is going to dig up his code for us.
> 
> ------- start of cut text --------------
> From: Mike Brzozowski
> To: Henry Stern
> Cc: Daniel Quinlan, Mike Brzozowski, Andrew Y. Ng, Rajat Raina, David Willcox,
>     Thomas Trappenberg
> Subject: Re: initial findings
> 
> Hi everyone,
> 
> After rerunning our logistic regression trial with 90-10 splits I was
> able to obtain similar values:
> 
> (threshold = 0: min confidence of spam = 50%)
> false positive: mean=0.149% std=0.157%
> false negative: mean=0.559% std=0.211%
> 
> (threshold = -1: min confidence of spam = 27%)
> false positive: mean=0.238% std=0.193%
> false negative: mean=0.330% std=0.210%
> 
> LR seems to be more resilient to the bias in the feature set towards
> spam features if we allow negative weights; in test runs an average of
> 26% of the features had negative weights; 73% had positive weights.
> 
> We also tried applying a "cost" to make misclassifying ham more
> expensive. This takes the form of modifying the objective function to
> scale the certainty of ham classifications by a constant factor
> /alpha/. With alpha = 0.25 we've been able to derive similar results:
> 
> (threshold = 0: min confidence of spam = 50%)
> false positive: mean=0% std=0%
> false negative: mean=0.619% std=0.311%
> 
> I've attached an FP-FN plot comparing logistic regression for various
> alphas to the numbers in cvs for SA 3.0's feature training. We're still
> looking into why alpha's effect is non-monotonic.
> 
> Do you have any older data sets? We were hoping to do a longitudinal
> study on adversarial spam techniques.
> 
> Thanks,
> Mike
> ------- end ----------------------------
> 
> -- 
> Daniel Quinlan
> http://www.pathname.com/~quinlan/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDQyikMJF5cimLx9ARAgtLAJwIkc4pcYOdHzWaoC/ImA6XddLuQgCfRlJk
iXVRx/SqqSfZmOF3l0vV4CI=
=bjgC
-----END PGP SIGNATURE-----