You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jatinpreet Singh (JIRA)" <ji...@apache.org> on 2014/09/20 17:38:34 UTC
[jira] [Updated] (SPARK-3614) Filter on minimum occurrences of a
term in IDF
[ https://issues.apache.org/jira/browse/SPARK-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jatinpreet Singh updated SPARK-3614:
------------------------------------
Description:
The IDF class in MLlib does not provide the capability of defining a minimum number of documents a term should appear in the corpus. The idea is to have a cutoff variable which defines this minimum occurrence value, and the terms which have lower frequency are ignored.
Mathematically,
IDF(t,D)=(log|D|+1) / (DF(t,D)+1), for DF(t,D) >=minimumOccurance
where,
D is the total number of documents in the corpus
DF(t,D) is the number of documents that contains term t
minimumOccurance is the minimum number of documents the term appears in the document corpus
This would have an impact on accuracy as terms that appear in less than a certain limit of documents, have low or no importance in TFIDF vectors.
was:
The IDF class in MLlib does not provide the capability of defining a minimum number of documents a term should appear in the corpus. The idea is to have a cutoff variable which defines this minimum occurrence value, and the terms which have lower frequency are ignored.
Mathematically,
IDF(t,D)=(log|D|+1) / (DF(t,D)+1), for DF(t,D) >=minimumOccurance
where,
|D| is the total number of documents in the corpus
DF(t,D) is the number of documents that contains term t
minimumOccurance is the minimum number of documents the term appears in the document corpus
This would have an impact on accuracy as terms that appear in less than a certain limit of documents, have low or no importance in TFIDF vectors.
> Filter on minimum occurrences of a term in IDF
> -----------------------------------------------
>
> Key: SPARK-3614
> URL: https://issues.apache.org/jira/browse/SPARK-3614
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Jatinpreet Singh
> Priority: Minor
> Labels: TFIDF
>
> The IDF class in MLlib does not provide the capability of defining a minimum number of documents a term should appear in the corpus. The idea is to have a cutoff variable which defines this minimum occurrence value, and the terms which have lower frequency are ignored.
> Mathematically,
> IDF(t,D)=(log|D|+1) / (DF(t,D)+1), for DF(t,D) >=minimumOccurance
> where,
> D is the total number of documents in the corpus
> DF(t,D) is the number of documents that contains term t
> minimumOccurance is the minimum number of documents the term appears in the document corpus
> This would have an impact on accuracy as terms that appear in less than a certain limit of documents, have low or no importance in TFIDF vectors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org