[jira] [Resolved] (SPARK-3614) Filter on minimum occurrences of a term in IDF

["Xiangrui Meng (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiangrui Meng resolved SPARK-3614.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.0

Issue resolved by pull request 2494
[https://github.com/apache/spark/pull/2494]

> Filter on minimum occurrences of a term in IDF 
> -----------------------------------------------
>
>                 Key: SPARK-3614
>                 URL: https://issues.apache.org/jira/browse/SPARK-3614
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>            Reporter: Jatinpreet Singh
>            Assignee: RJ Nowling
>            Priority: Minor
>              Labels: TFIDF
>             Fix For: 1.2.0
>
>
> The IDF class in MLlib does not provide the capability of defining a minimum number of documents a term should appear in the corpus. The idea is to have a cutoff variable which defines this minimum occurrence value, and the terms which have lower frequency are ignored.
> Mathematically,
> IDF(t,D)=log( (|D|+1)/(DF(t,D)+1) ), for DF(t,D) >=minimumOccurance
> where, 
> D is the total number of documents in the corpus
> DF(t,D) is the number of documents that contain the term t
> minimumOccurance is the minimum number of documents the term appears in the document corpus
> This would have an impact on accuracy as terms that appear in less than a certain limit of documents, have low or no importance in TFIDF vectors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org