You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Yu Fu <yu...@umbc.edu> on 2011/04/07 11:40:55 UTC
JAMES-1216 [gsoc2011] Design and implement machine learning filters
and categorization for mail
Hi Everyone,
This is Vicki. I am applying to Gsoc 2011 on JAMES-1216[jira].
I have the question for updating Bayesian Analysis to 3.0. Because the
container switch from Avalon to Spring, some interfaces need to rewrite? Is
anyone know how it should be?
I have attached my proposal and the figure here.I think it should act as the
starting point for me to contribute to James community. Welcome to any
feedback.
_______________________________________________________________________________________________________________
*Proposal Title: *Design and implement machine learning filters and
categorization for anti spam in the mail system
*Student Name:* Yu Fu
*Student E-mail: *yufu1@umbc.edu
*Organization/Project*: The Apache Software Foundation
*Assigned Mentor*: Eric Charles, Robert Burrell Donkin
Proposal Abstract: Using advanced machine learning methods K-nearest
neighbors to implement intelligent filter mail feature, To strengthen
anti-spam capability on the beyond of current Bayesian Analysis method.
*Detailed Description:*
*1. Understand James Anti-Spam Features*
Current James Anti-Spam have three parts, which I marked with the red in the
figure below. The figure also shows the whole relationship for the basic
concepts.
� The first part *fastfail* in this SMTPServer is to reject a
message on the smtplevel if a configured hits amount is reached. It operates
in the background of a machine, listening on a specific TCP port. In this
level, the server will filter non-existed users, DSN filter, domains with
invalid MX record, ignore duplicated recipients. Also IP Addresses which
send emails to the configured recipients will get blacklisted for the
configured time. The configuration for the certain port and hits score can
be set in the smtpserver.xml.
� *SpamAssassin* in the Mailet is to classify the messages as spam
or not with an internal (configurable) score threshold. Usually a message
will only be considered as spam if it matches multiple criteria; matching
just a single test will not usually be enough to reach the threshold.
� *BayesianAnalysis* in the Mailet are based on the Bayesian
Probably to classify to spam or not. It relied on the training data from the
users� judgment. Users need to manually judge as spam and send to
spam@thisdomain.com, oppositely, if judge as non-spam and then send to
not.spam@thisdomain.com. BayesianAnalysisfeeder learn from these training
dataset, and build predictive models on Bayesian Probably. There will be a
certain table for maintaining the frequency of Corpus for keywords in the
database. Every 10 mins a thread in the BayesianAnalysis will check and
update the table. Also the correct approach is to send the original spam or
non-spam as an attachment to another message send to the feeder in order to
avoid the bias from the current sender�s email header.
*2. Motivation*
Navie Bayes approach, which is based on Bayesian Probablity, is the most
popular method. But for some too popular Phrase or keyword, the Navie Bayes
cannot work well. Consider �Click Here to Unsubscribe� Phrase occurs 10
times as often in spam as good. P(click|spam) is 10 times higher, and
P(unsubscribe|spam) is 10 times higher Multiply together, get factor of
1000. The bias of these hot keywords will heavily hurt the accuracy of the
filter.
I want to implement K-nearest neighbor, which has much better experiment [1]
result than our current approach, and The training time of KNN is very fast.
This project will be the start point for me to contribute to the James Open
Source community. I treat Anti-Spam filter is a practical issue rather than
a theoretical tech problem. In this community we need to win with abandon
practice and the strong analysis reasoning. I am enjoying implement
different ways to find most efficient and easy method.
*3. Project Scope*
The project will be based on the James Server 3.0 framework, and it will
restrict the categories to spam/not spam.
� Upgrade Bayesian Analysis from 2.3 to 3.0
� Implement �TF-IDF� and �binary Vector� feature selection.
� Implement K-nearest neighbor algorithm to sever as the new
Anti-Spam filter.
*4. Approach*
My idea is use K-nearest neighbor algorithm to classify the mail to spam or
non-spam. Computing space distance between different text words, KNN has
better experimental performance considering the accuracy measurement [1].
*Feature Selection*
Both of the �TF-IDF� (case insensitive) and �binary Vector� (case
insensitive) feature selection will be implemented.
*TF-IDF*: The tf�idf weight (term frequency�inverse document frequency) is
is a statistical measure used to evaluate how important a word is to a
document <http://en.wikipedia.org/wiki/Document> in a collection or
corpus<http://en.wikipedia.org/wiki/Text_corpus>.
The importance increases
proportionally<http://en.wikipedia.org/wiki/Proportionality_(mathematics)>
to
the number of times a word appears in the document but is offset by the
frequency of the word in the corpus. Variations of the tf�idf weighting
scheme are often used by search
engines<http://en.wikipedia.org/wiki/Search_engine> as
a central tool in scoring and ranking a document's
relevance<http://en.wikipedia.org/wiki/Relevance_(information_retrieval)>
given
a user query <http://en.wikipedia.org/wiki/Information_retrieval>. So it can
avoid the bias on Bayesian approach. The variable for TF-IDF will be tuned
by the experiment, and also can manual be configured by users.
Weighted *TF-IDF*: based TF-IDF, we can set different weighted, considering
the words position in the mail structure. I have assumption: the words in
message header are more important than the body.
*Binary Vector*: it maintains the information of each feature (term) exist
or not (1 or 0) for each training data. The main variable here is the size
of vector, we will set and consider with memory size for this case.
*Variables*: Vector Size, Ratio for TF-IDF. Weight for words position
*Feature extraction *I will use mutual information in order to reduce
dimension. Using Mutual information of each word with the class variable has
been used to select important features. Mutual information of a word
captures the amount of information gained in classifying documents by
knowing the presence and absence of the word [3].
*KNN:* this method will still feed the predictive model on the training
data, which can share the training data from Bayesian Analysis.The
k-nearest neighbor�s algorithm (k-NN) is a method for
classifying <http://en.wikipedia.org/wiki/Statistical_classification> objects
based on closest training examples in the feature
space<http://en.wikipedia.org/wiki/Feature_space>.
The k-nearest neighbor algorithm is amongst the simplest of all machine
learning <http://en.wikipedia.org/wiki/Machine_learning> algorithms: an
object is classified by a majority vote of its neighbors, with the object
being assigned to the class most common amongst its k nearest neighbors
(k is a positive integer <http://en.wikipedia.org/wiki/Integer>, typically
small). Here K=2, because we need to classier for spam or not-spam.
KNN method implements:
� Method 1: To create a �binary vector� which maintain
the information of each feature (term) exist or not (1 or 0) for each
training data. For each unclassified mail to create a binary vector too,
then using cosine distance to find out the top 2 closest training data from
the then find out which class is the unclassified mail belongs to. Compute
Euclidean distance from target plot to those that were sampled. Order
samples taking for account calculated distances. Choose heuristically
optimal 2 nearest neighbor based on RMSE
<http://en.wikipedia.org/wiki/RMSE> done
by cross validation technique. Calculate an inverse distance weighted
average with the k-nearest multivariate neighbors.
� Method 2: Use �TF-IDF� respectively instead of binary
Vector in Method 1.
The enhanced feature for Bayesian method:
1. Filtering for certain words based on the Bayesian probability of
single words.
2. Filtering for all the words based on the combination probability of
all the words.
3. Filtering for the rare words based on the Bayesian probability.
Schedule
1 week to upgrade Bayesian Analysis feature to 3.0.
2 weeks implement Parser to selection method (3 weeks implement and 1 week
test)
3 weeks implement KNN method (2 weeks implement and 1 week test)
2 weeks tune the predictive model with different parameters.
1 week finish the document and guide for using and tuning.
[1] Anti-Spam Filter Based on Na�ve Bayes, SVM, and KNN model. *
irlab.csie.ntu.edu.tw/~chean/perWeb/projects/AI/AI_final.pdf*
**[2] An Empirical Performance Comparison of Machine Learning Methods for
Spam E-Mail Categorization.
http://www.computer.org/portal/web/csdl/doi/10.1109/ICHIS.2004.21
<http://www.computer.org/portal/web/csdl/doi/10.1109/ICHIS.2004.21>[3] Text
Categorization Using Weight Adjusted k-Nearest Neighbor Classification
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.109.1259&rep=rep1&type=pdf
--
Yu Fu
yufu1@umbc.edu
443-388-6654
Re: JAMES-1216 [gsoc2011] Design and implement machine learning filters
and categorization for mail
Posted by Eric Charles <er...@apache.org>.
> PS: I left some comment on google melange regarding your application (we
> are today in the last mile).
Hi Vicky,
It seems google-melange does not let see my comments to students.
I publish them here (hoping you can still process some/all of them
before tomorrow)
- You talk about Mime4J parser: I suppose it will be used to parse the
mail (maybe syntax of the sentence can be ameliorated)
- About Feature Selection, you can read/integrate Robert's comments on
https://issues.apache.org/jira/browse/JAMES-1216
- You can "open the door" saying that the project will categorize with
spam/notspam, but that it could be extended further (out-of-scope of the
project) to other categories.
- Will the end-user still be obliged to send to spam@domain.com or
notspam@domain.com to train the system?
- irlab.csie.ntu.edu.tw/~chean/perWeb/projects/AI/AI_final.pdf misses
http:// in front (even with http://, it does not respond)
- What about your bio? (not the details, simply the
references/experiences with weka,..)
Tks,
- Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: JAMES-1216 [gsoc2011] Design and implement machine learning filters
and categorization for mail
Posted by Eric Charles <er...@apache.org>.
Hi Vicky,
Welcome!
Hope you will enjoy some time with James community, and especially with
Robert and I who will mentor this.
Further to container/database access change between 2.3 and 3.0, there
are sometimes stuff that need to be tuned.
I should look at the Bayesian mailets in James 3.0 before end of this week.
In the meantime, you can try to build James from source
(http://james.apache.org/server/3/dev-build.html), configure (launch
jConsole, and go on MBeans tab), and run with standard configuration
(use your favorite mail client to send/receive mails).
Come back on this mailing list if you encounter any issue on this (the
doc is not 100% in line with the src trunk, we are working on it).
If everything goes well, you can also try the Bayesian mailets
(http://wiki.apache.org/james/Bayesian_Analysis), and maybe have
exceptions with SQL queries towards the embedded derby database.
Those mailets still use the SQL queries you will find in
sqlResources.xml (the section
"org.apache.james.util.bayesian.JDBCBayesianAnalyzer").
All databases was done in 2.3 via this file, we tend in 3.0 to implement
database via OpenJPA.
For the project, we will move the needed classes to a sandbox (see
http://markmail.org/message/ichoc67sz4sjybig)
Tks again,
PS: I left some comment on google melange regarding your application (we
are today in the last mile).
On 7/04/2011 11:40, Yu Fu wrote:
> Hi Everyone,
> This is Vicki. I am applying to Gsoc 2011 on JAMES-1216[jira].
> I have the question for updating Bayesian Analysis to 3.0. Because the
> container switch from Avalon to Spring, some interfaces need to rewrite?
> Is anyone know how it should be?
>
> I have attached my proposal and the figure here.I think it should act as
> the starting point for me to contribute to James community. Welcome to
> any feedback.
>
> _______________________________________________________________________________________________________________
>
> *Proposal Title: *Design and implement machine learning filters and
> categorization for anti spam in the mail system
> *Student Name:*Yu Fu
> *Student E-mail: *yufu1@umbc.edu <ma...@umbc.edu>
> *Organization/Project*: The Apache Software Foundation
> *Assigned Mentor*: Eric Charles, Robert Burrell Donkin
> Proposal Abstract: Using advanced machine learning methods K-nearest
> neighbors to implement intelligent filter mail feature, To strengthen
> anti-spam capability on the beyond of current Bayesian Analysis method.
> *Detailed Description:*
>
>
> *1. Understand James Anti-Spam Features*
>
> Current James Anti-Spam have three parts, which I marked with the red in
> the figure below. The figure also shows the whole relationship for the
> basic concepts.
>
> ·The first part *fastfail* in this SMTPServer is to reject a message on
> the smtplevel if a configured hits amount is reached. It operates in the
> background of a machine, listening on a specific TCP port. In this
> level, the server will filter non-existed users, DSN filter, domains
> with invalid MX record, ignore duplicated recipients. Also IP Addresses
> which send emails to the configured recipients will get blacklisted for
> the configured time. The configuration for the certain port and hits
> score can be set in the smtpserver.xml.
>
> ·*SpamAssassin*in the Mailet is to classify the messages as spam or not
> with an internal (configurable) score threshold. Usually a message will
> only be considered as spam if it matches multiple criteria; matching
> just a single test will not usually be enough to reach the threshold.
>
> ·*BayesianAnalysis*in the Mailet are based on the Bayesian Probably to
> classify to spam or not. It relied on the training data from the users’
> judgment. Users need to manually judge as spam and send to
> spam@thisdomain.com <ma...@thisdomain.com>, oppositely, if judge
> as non-spam and then send to not.spam@thisdomain.com
> <ma...@thisdomain.com>. BayesianAnalysisfeeder learn from
> these training dataset, and build predictive models on Bayesian
> Probably. There will be a certain table for maintaining the frequency of
> Corpus for keywords in the database. Every 10 mins a thread in the
> BayesianAnalysis will check and update the table. Also the correct
> approach is to send the original spam or non-spam as an attachment to
> another message send to the feeder in order to avoid the bias from the
> current sender’s email header.
>
> *2. Motivation*
>
> Navie Bayes approach, which is based on Bayesian Probablity, is the most
> popular method. But for some too popular Phrase or keyword, the Navie
> Bayes cannot work well. Consider “Click Here to Unsubscribe” Phrase
> occurs 10 times as often in spam as good. P(click|spam) is 10 times
> higher, and P(unsubscribe|spam) is 10 times higher Multiply together,
> get factor of 1000. The bias of these hot keywords will heavily hurt the
> accuracy of the filter.
>
> I want to implement K-nearest neighbor, which has much better experiment
> [1] result than our current approach, andThe training time of KNN is
> very fast.
>
> This project will be the start point for me to contribute to the James
> Open Source community. I treat Anti-Spam filter is a practical issue
> rather than a theoretical tech problem. In this community we need to win
> with abandon practice and the strong analysis reasoning. I am enjoying
> implement different ways to find most efficient and easy method.
>
> *3. Project Scope*
> The project will be based on the James Server 3.0 framework, and it will
> restrict the categories to spam/not spam.
>
> ·Upgrade Bayesian Analysis from 2.3 to 3.0
>
> ·Implement “TF-IDF” and “binary Vector” feature selection.
>
> ·Implement K-nearest neighbor algorithmto sever as the new Anti-Spam filter.
>
>
> *4. Approach*
>
> My idea is use K-nearest neighbor algorithm to classify the mail to spam
> or non-spam. Computing space distance between different text words, KNN
> has better experimental performance considering the accuracy measurement
> [1].
>
> *Feature Selection*
>
> Both of the “TF-IDF” (case insensitive) and “binary Vector” (case
> insensitive) feature selection will be implemented.
>
> *TF-IDF*: The tf–idf weight (term frequency–inverse document frequency)
> is is a statistical measure used to evaluate how important a word is to
> a document <http://en.wikipedia.org/wiki/Document> in a collection or
> corpus <http://en.wikipedia.org/wiki/Text_corpus>. The importance
> increases proportionally
> <http://en.wikipedia.org/wiki/Proportionality_(mathematics)> to the
> number of times a word appears in the document but is offset by the
> frequency of the word in the corpus. Variations of the tf–idf weighting
> scheme are often used by search engines
> <http://en.wikipedia.org/wiki/Search_engine> as a central tool in
> scoring and ranking a document's relevance
> <http://en.wikipedia.org/wiki/Relevance_(information_retrieval)> given a
> user query <http://en.wikipedia.org/wiki/Information_retrieval>. So it
> can avoid the bias on Bayesian approach. The variable for TF-IDF will be
> tuned by the experiment, and also can manual be configured by users.
>
> Weighted *TF-IDF*: based TF-IDF, we can set different weighted,
> considering the words position in the mail structure. I have assumption:
> the words in message header are more important than the body.
>
> *Binary Vector*: it maintains the information of each feature (term)
> exist or not (1 or 0) for each training data. The main variable here is
> the size of vector, we will set and consider with memory size for this case.
>
> *Variables*: Vector Size, Ratio for TF-IDF. Weight for words position
>
> *Feature extraction *I will use mutual information in order to reduce
> dimension. Using Mutual information of each word with the class variable
> has been used to select important features. Mutual information of a word
> captures the amount of information gained in classifying documents by
> knowing the presence and absence of the word [3].
>
> *KNN:*this method will still feed the predictive model on the training
> data, which can share the training data from Bayesian
> Analysis.The k-nearest neighbor’s algorithm (k-NN) is a method for
> classifying
> <http://en.wikipedia.org/wiki/Statistical_classification> objects based
> on closest training examples in the feature space
> <http://en.wikipedia.org/wiki/Feature_space>. The k-nearest neighbor
> algorithm is amongst the simplest of all machine learning
> <http://en.wikipedia.org/wiki/Machine_learning> algorithms: an object is
> classified by a majority vote of its neighbors, with the object being
> assigned to the class most common amongst its k nearest neighbors (k is
> a positive integer <http://en.wikipedia.org/wiki/Integer>, typically
> small). Here K=2, because we need to classier for spam or not-spam.
>
> KNN method implements:
>
> ·Method 1: To create a “binary vector” which maintain the information of
> each feature (term) exist or not (1 or 0) for each training data. For
> each unclassified mail to create a binary vector too, then using cosine
> distance to find out the top 2 closest training data from the then find
> out which class is the unclassified mail belongs to. Compute Euclidean
> distance from target plot to those that were sampled. Order samples
> taking for account calculated distances. Choose heuristically
> optimal 2 nearest neighbor based on RMSE
> <http://en.wikipedia.org/wiki/RMSE> done by cross validation technique.
> Calculate an inverse distance weighted average with the k-nearest
> multivariate neighbors.
>
> ·Method 2: Use “TF-IDF” respectively instead of binary Vector in Method 1.
>
> The enhanced feature for Bayesian method:
>
> 1. Filtering for certain words based on the Bayesian probability of
> single words.
> 2. Filtering for all the words based on the combination probability
> of all the words.
> 3. Filtering for the rare words based on the Bayesian probability.
>
> Schedule
>
> 1 week to upgrade Bayesian Analysis feature to 3.0.
>
> 2 weeks implement Parser to selection method (3 weeks implement and 1
> week test)
>
> 3 weeks implement KNN method (2 weeks implement and 1 week test)
>
> 2 weeks tune the predictive model with different parameters.
>
> 1 week finish the document and guide for using and tuning.
>
> [1]Anti-Spam Filter Based on Naïve Bayes, SVM, and KNN model.
> _irlab.csie.ntu.edu.tw/~chean/perWeb/projects/AI/AI_final.pdf
> <http://irlab.csie.ntu.edu.tw/~chean/perWeb/projects/AI/AI_final.pdf>_
>
> __[2] An Empirical Performance Comparison of Machine Learning Methods
> for Spam E-Mail Categorization.
> http://www.computer.org/portal/web/csdl/doi/10.1109/ICHIS.2004.21
>
> <http://www.computer.org/portal/web/csdl/doi/10.1109/ICHIS.2004.21>[3]
> Text Categorization Using Weight Adjusted k-Nearest Neighbor
> Classification
> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.109.1259&rep=rep1&type=pdf
> <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.109.1259&rep=rep1&type=pdf>
>
>
>
>
>
> --
> Yu Fu
> yufu1@umbc.edu <ma...@umbc.edu>
> 443-388-6654
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org