stop word-lists VS maxDFPercent

[David Noel <da...@gmail.com>]

What's everyone's opinion on using large stop word lists vs a very
small value for maxDFPercent (like 30)? I'm playing around with both
and am having trouble deciding whether one is better than the other,
or if I should use a combination of both. My data set is one day's
worth of news articles gathered from 1000 online news outlets. It's
probably similar to the reuters data set, but with a little more
noise. I used Boilerpipe for article extraction.

I spent a good while Googling around to build the largest (English)
stop word-list I could. I'll paste it below for anyone who's
interested and would like to save themselves an hour of Googling and
collating.

-------------------------------------------------------------

'll,'tis,'twas,a,a's,aan,able,about,above,abroad,abst,accordance,according,accordingly,across,act,actually,ad,added,adj,af,affected,affecting,affects,after,afterwards,again,against,ago,ah,ahead,ain't,al,all,alle,alles,allow,allows,almost,alone,along,alongside,already,als,also,alt,although,altijd,always,am,amid,amidst,among,amongst,amoungst,amount,an,and,anden,andere,announce,another,any,anybody,anyhow,anymore,anyone,anything,anyway,anyways,anywhere,apart,apparently,appear,appreciate,appropriate,approximately,are,area,areas,aren,aren't,arent,arise,around,as,aside,ask,asked,asking,asks,associated,at,auth,available,away,awfully,b,back,backed,backing,backs,backward,backwards,be,became,because,become,becomes,becoming,been,before,beforehand,began,begin,beginning,beginnings,begins,behind,being,beings,believe,below,ben,beside,besides,best,better,between,beyond,big,bij,bill,biol,blev,blive,bliver,both,bottom,brief,briefly,but,by,c,c'mon,c's,ca,call,came,can,can't,cannot,cant,caption,case,cases,cause,causes,certain,certainly,changes,clear,clearly,co,co.,com,come,comes,computer,con,concerning,consequently,consider,considering,contain,containing,contains,corresponding,could,could've,couldn't,couldnt,course,cry,currently,d,da,daar,dan,dare,daren't,dat,date,de,dear,definitely,dem,den,denne,der,deres,describe,described,despite,det,detail,dette,deze,did,didn't,die,differ,different,differently,dig,din,directly,disse,dit,do,doch,doen,does,doesn't,dog,doing,don,don't,done,door,down,downed,downing,downs,downwards,du,due,during,dus,e,each,early,ed,edu,een,eens,effect,efter,eg,eight,eighty,either,eleven,eller,else,elsewhere,empty,en,end,ended,ending,ends,enough,entirely,er,especially,et,et-al,etc,even,evenly,ever,evermore,every,everybody,everyone,everything,everywhere,ex,exactly,example,except,f,face,faces,fact,facts,fairly,far,farther,felt,few,fewer,ff,fifteen,fifth,fify,fill,find,finds,fire,first,five,fix,followed,following,follows,for,forever,former,formerly,forth,forty,forward,found,four,fra,from,front,full,fully,further,furthered,furthering,furthermore,furthers,g,gave,ge,geen,general,generally,get,gets,getting,geweest,give,given,gives,giving,go,goes,going,gone,good,goods,got,gotten,great,greater,greatest,greeting,greetings,group,grouped,grouping,groups,h,haar,had,hadn't,half,ham,han,hans,happen,happens,har,hardly,has,hasn't,hasnt,havde,have,haven't,having,he,he'd,he'll,he's,heb,hebben,hed,heeft,hello,help,hem,hence,hende,hendes,her,here,here's,hereafter,hereby,herein,heres,hereupon,hers,herself,hes,het,hi,hid,hier,high,higher,highest,hij,him,himself,his,hither,hoe,home,hopeful,hopefully,hos,how,how'd,how'll,how's,howbeit,however,hun,hundred,hvad,hvis,hvor,i,i'd,i'll,i'm,i've,id,ie,iemand,iets,if,ignored,ik,ikke,im,immediate,immediately,importance,important,in,inasmuch,inc,inc.,ind,indeed,index,indicate,indicated,indicates,information,inner,inside,insofar,instead,interest,interested,interesting,interests,into,invention,inward,is,isn't,it,it'd,it'll,it's,itd,its,itself,j,ja,je,jeg,jer,jo,just,k,kan,keep,keeps,kept,kg,kind,km,knew,know,knowing,known,knows,kon,kunne,kunnen,l,large,largely,last,lastly,late,lately,later,latest,latter,latterly,least,less,lest,let,let's,lets,like,liked,likely,likewise,line,little,long,longer,longest,look,looking,looks,low,lower,ltd,m,maar,made,mainly,make,makes,making,man,mange,many,may,maybe,mayn't,me,mean,means,meantime,meanwhile,med,meer,meget,member,members,men,merely,met,mg,mig,might,might've,mightn't,mij,mijn,mill,million,millions,min,mine,minus,miss,mit,ml,mod,moet,more,moreover,most,mostly,move,mr,mrs,much,must,must've,mustn't,my,myself,n,na,naar,name,namely,nay,nd,near,nearly,necessarily,necessary,ned,need,needed,needing,needn't,needs,neither,never,neverf,neverless,nevertheless,new,newer,newest,next,niet,niets,nine,ninety,no,no-one,nobody,nog,noget,nogle,non,none,nonetheless,noone,nor,normally,nos,not,noted,nothing,notwithstanding,novel,now,nowhere,nu,number,numbers,o,obtain,obtained,obviously,of,off,often,og,ogs,oh,ok,okay,old,older,oldest,om,omdat,omitted,on,once,onder,one,one's,ones,only,ons,onto,ook,op,open,opened,opening,opens,opposite,or,ord,order,ordered,ordering,orders,os,other,others,otherwise,ought,oughtn't,our,ours,ourselves,out,outside,over,overall,owing,own,p,page,pages,part,parted,particular,particularly,parting,parts,past,per,perhaps,place,placed,places,please,plus,point,pointed,pointing,points,poorly,possible,possibly,potentially,pp,predominantly,present,presented,presenting,presents,presumably,presume,presumed,previously,primarily,primary,probable,probably,problem,problems,prompt,promptly,proud,provide,provided,provides,put,puts,q,que,quick,quickly,quite,qv,r,ran,rather,rd,re,readily,really,reasonable,reasonably,recent,recently,reeds,ref,refs,regard,regarding,regardless,regards,relate,related,relative,relatively,respective,respectively,result,resulted,resulting,results,right,room,rooms,round,run,s,said,same,saw,say,saying,says,sec,second,secondly,seconds,see,seeing,seem,seemed,seeming,seems,seen,sees,self,selv,selves,sensible,sent,serious,seriously,seven,several,shall,shan't,she,she'd,she'll,she's,shed,shes,should,should've,shouldn't,show,showed,showing,shown,showns,shows,side,sides,sig,significant,significantly,similar,similarly,sin,since,sincere,sine,sit,six,sixty,skal,skulle,slightly,small,smaller,smallest,so,som,some,somebody,someday,somehow,someone,somethan,something,sometime,sometimes,somewhat,somewhere,soon,sorry,specifically,specified,specify,specifying,state,states,still,stop,strongl,strongly,sub,substantial,substantially,successfully,such,sufficient,sufficiently,suggest,suggested,suggests,sup,sure,system,t,t's,take,taken,taking,te,tegen,tell,tells,ten,tends,th,than,thank,thanks,thanx,that,that'll,that's,that've,thats,the,their,theirs,them,themselves,then,thence,there,there'd,there'll,there're,there's,there've,thereafter,thereby,therefore,therein,theres,thereupon,these,they,they'd,they'll,they're,they've,thi,thick,thin,thing,things,think,thinks,third,thirty,this,thorough,thoroughly,those,though,thought,thoughts,three,through,throughout,thru,thus,til,till,tis,to,toch,today,toen,together,too,took,top,tot,toward,towards,tried,tries,truly,try,trying,turn,turned,turning,turns,twas,twelve,twenty,twice,two,u,ud,uit,un,under,underneath,undoing,unfortunately,unless,unlike,unlikely,until,unto,up,upon,upwards,us,use,used,useful,uses,using,usually,uucp,uw,v,value,van,var,various,veel,versus,very,vi,via,vil,ville,viz,voor,vor,vs,være,været,w,want,wanted,wanting,wants,waren,was,wasn't,wat,way,ways,we,we'd,we'll,we're,we've,welcome,well,wells,went,werd,were,weren't,wezen,what,what'd,what'll,what's,what've,whatever,when,when'd,when'll,when's,whence,whenever,where,where'd,where'll,where's,whereafter,whereas,whereby,wherein,whereupon,wherever,whether,which,whichever,while,whilst,whither,who,who'd,who'll,who's,whoever,whole,whom,whomever,whose,why,why'd,why'll,why's,wie,wil,will,willing,wish,with,within,without,won't,wonder,worden,wordt,work,worked,working,works,would,would've,wouldn't,www,x,y,year,years,yes,yet,you,you'd,you'll,you're,you've,young,younger,youngest,your,yours,yourself,yourselves,z,zal,ze,zelf,zero,zich,zij,zijn,zo,zonder,zou

Re: stop word-lists VS maxDFPercent

[David Noel <da...@gmail.com>]

> I spent a good while Googling around to build the largest (English)
> stop word-list I could. I'll paste it below for anyone who's
> interested and would like to save themselves an hour of Googling and
> collating.

...cleaned it up some more:

'll,'tis,'twas,a,a's,able,about,above,accordance,according,accordingly,across,act,acts,acted,acting,actually,add,adds,added,adding,adj,affect,affected,affecting,affects,after,afterwards,again,against,ago,ah,ahead,ain't,al,all,allow,allows,allowing,almost,alone,along,alongside,already,also,alt,although,always,am,amid,amidst,among,amongst,amoungst,amount,amounts,amounted,an,and,announce,announced,announcing,another,anothers,any,anybodies,anybody,anyhow,anymore,anyone,anyone's,anything,anything's,anyway,anyways,anywhere,apart,apparent,apparently,appear,appeared,appearing,appreciate,appreciated,appropriate,appropriately,approximate,approximately,are,area,areas,area's,aren,aren't,arent,arise,arisen,around,as,aside,ask,asked,asking,asks,associated,at,auth,available,away,awful,awfully,b,back,backed,backing,backs,backward,backwards,be,became,because,become,becomes,becoming,been,before,beforehand,began,begin,begins,beginning,beginnings,behind,being,beings,believe,believed,below,beside,besides,best,better,between,beyond,big,both,bottom,brief,briefly,but,by,c,c'mon,c's,call,came,can,can't,cannot,cant,caption,captions,captioned,case,cases,cause,caused,causes,certain,certainly,change,changes,clear,clearly,co,co.,com,come,comes,computer,computers,concern,concerning,consequent,consequently,consider,considered,considering,contain,contained,containing,contains,correspond,corresponding,could,could've,couldn't,couldnt,course,cry,current,currently,dare,daren't,dat,date,dates,dated,dear,dearly,definite,definitely,describe,described,describing,despite,detail,details,did,didn't,die,died,differ,differs,different,differently,dig,digs,din,direct,directly,do,does,doesn't,dog,doin,doing,don't,done,door,down,downed,downing,downs,downwards,due,during,e,each,early,edu,effect,eg,eight,eights,eighty,either,eleven,else,elsewhere,empty,en,end,ended,ending,ends,enough,entire,entirely,er,especially,et,et-al,etc,even,evens,evened,evenly,ever,evermore,every,everybody,everybody's,everyone,everyone's,everything,everywhere,ex,exact,exactly,example,except,excepted,excepting,f,face,faced,faces,fact,facts,factual,fair,fairly,far,farther,felt,few,fewer,ff,fifteen,fifteenth,fifth,fify,fill,filled,filling,fills,find,finds,finding,fire,fires,first,five,fix,fixing,fixed,fixes,follow,followed,follower,followers,following,follows,for,forever,former,formerly,forth,forthly,forty,forward,forwards,forwarding,found,four,from,front,fronted,fronting,full,fully,further,furthered,furthering,furthermore,furthers,g,gave,general,generals,generally,get,gets,getting,give,given,gives,giving,go,goes,going,gone,good,goods,got,gotten,great,greatly,greater,greatest,greeting,greetings,group,grouped,grouping,groups,h,had,hadn't,half,ham,happen,happened,happens,har,hardly,has,hasn't,hasnt,havde,have,haven't,having,he,he'd,he'll,he's,heb,hed,hello,help,helps,helped,hem,hems,hence,her,here,here's,hereafter,hereby,herein,heres,hereupon,hers,herself,hes,hi,hid,hier,high,highs,highly,higher,highest,him,himself,his,hither,hoe,hoes,home,hopeful,hopefully,hopefuls,hos,how,how'd,how'll,how's,howbeit,however,hun,hundred,i,i'd,i'll,i'm,i've,id,ie,iemand,if,ignored,im,immediate,immediately,importance,important,in,inasmuch,inc,inc.,indeed,index,indexes,indexed,indexing,indicate,indicated,indicates,information,informations,inner,inside,insofar,instead,interest,interests,interested,interesting,into,invention,inventions,inward,inwards,is,isn't,it,it'd,it'll,it's,itd,its,itself,j,just,k,keep,keeps,kept,keeping,kind,kinds,km,knew,know,knowing,known,knows,l,large,largely,last,lasts,lastly,late,lately,later,latest,latter,latterly,least,less,lest,let,let's,lets,like,liked,liking,likely,likewise,line,lined,little,long,longer,longing,longest,look,looking,looks,looked,low,lows,lower,lowering,lowered,ltd,m,made,mainly,make,makes,making,man,mans,mange,many,may,maybe,maybe's,mayn't,me,mean,meaning,meant,means,meantime,meanwhile,member,members,men,mens,mere,merely,met,might,might've,mightn't,mill,million,millions,min,mins,mine,mined,mining,minus,miss,misses,missing,mod,more,moreover,most,mostly,move,mr,mrs,much,must,must've,mustn't,my,myself,n,na,name,named,naming,namely,nay,nd,near,neared,nearly,necessarily,necessary,need,needed,needing,needn't,needs,neither,never,neverless,nevertheless,new,newer,newest,next,nine,ninety,no,no-one,nobody,non,none,nonetheless,noone,nor,normal,normally,not,noted,noting,nothing,nothing's,nothings,notwithstanding,novel,novels,now,nowhere,number,numbers,o,obtain,obtains,obtaining,obtained,obviously,of,off,often,oh,ok,okay,old,older,oldest,omitted,on,once,onder,one,one's,ones,only,onto,ook,open,opened,opening,opens,opposite,or,order,ordered,ordering,orders,other,others,otherwise,ought,oughtn't,our,ours,ourself,ourselves,out,outs,outside,over,overall,owe,owing,own,p,page,pages,paging,part,parted,particular,particularly,parting,parts,past,per,perhaps,place,placed,placing,places,please,pleased,plus,point,pointed,pointing,points,poor,poorly,possible,possibly,potentially,potential,predominantly,predominant,present,presented,presenting,presents,presumable,presumably,presume,presumed,previous,previously,primarily,primary,probable,probably,problem,problems,prompt,prompts,prompted,promptly,proud,provide,provided,provides,put,puts,q,que,quick,quickly,quite,qv,r,ran,rather,rd,re,readily,real,really,reasonable,reasonably,recent,recently,reeds,ref,refs,regard,regarding,regardless,regards,relate,related,relative,relatively,respective,respectively,result,resulted,resulting,results,right,rights,room,rooms,round,rounds,run,s,said,same,saw,say,saying,says,sec,second,secondly,seconds,see,seeing,seem,seemed,seeming,seems,seen,sees,self,selve,selves,sensible,sensibly,sent,serious,seriously,seven,seventy,several,shall,shan't,she,she'd,she'll,she's,shed,shes,should,should've,shouldn't,show,showed,showing,shown,showns,shows,side,sides,sig,significant,significantly,similar,similarly,since,sincere,sine,sit,six,sixty,slight,slightly,small,smaller,smallest,so,some,somebody,someday,somehow,someone,somethan,something,something's,sometime,sometimes,somewhat,somewhere,soon,sorry,specifically,specified,specify,specifying,state,states,stated,still,stills,stop,stops,strong,strongly,substantial,substantially,successful,successfully,such,sufficient,sufficiently,suggest,suggested,suggests,suggesting,sup,sups,sure,system,t,t's,take,taken,taking,tell,tells,telling,ten,tend,tends,tending,than,thank,thanks,thanx,thanking,that,that'll,that's,that've,thats,the,their,theirs,them,themselves,then,thence,there,there'd,there'll,there're,there's,there've,thereafter,thereby,therefore,therein,theres,thereupon,these,they,they'd,they'll,they're,they've,thick,thin,thins,thing,things,think,thinks,third,thirds,thirty,this,thorough,thoroughly,those,though,thought,thoughts,three,through,throughout,thru,til,till,tis,to,today,together,too,took,top,tot,toward,towards,tried,tries,truly,try,trying,turn,turned,turning,turns,twas,twelve,twenty,twice,two,u,ud,under,underneath,undo,undone,undid,undoing,unfortunate,unfortunately,unless,unlike,unliked,unlikely,until,unto,up,upon,upward,upwards,us,use,used,useful,uses,using,usually,uucp,v,value,various,versus,very,via,viz,vs,w,want,wanted,wanting,wants,was,wasn't,wat,way,ways,we,we'd,we'll,we're,we've,welcome,welcomes,welcomed,well,wells,went,werd,were,weren't,what,what'd,what'll,what's,whats,what've,whatever,when,when'd,when'll,when's,whence,whenever,where,where'd,where'll,where's,whereafter,whereas,whereby,wherein,whereupon,wherever,whether,which,whichever,while,whilst,whither,who,who'd,who'll,who's,whoever,whole,whom,whomever,whose,why,why'd,why'll,why's,will,willing,wish,with,within,without,won't,wonder,wondering,wonders,work,worked,working,works,would,would've,wouldn't,www,x,y,year,years,yes,yet,you,you'd,you'll,you're,you've,young,younger,youngest,your,yours,yourself,yourselves,z,zero,sdfsdfs

Re: stop word-lists VS maxDFPercent

[Ken Krugler <kk...@transpac.com>]

On May 12, 2014, at 6:24am, David Noel <da...@gmail.com> wrote:

> What's everyone's opinion on using large stop word lists vs a very
> small value for maxDFPercent (like 30)? I'm playing around with both
> and am having trouble deciding whether one is better than the other,
> or if I should use a combination of both.

Not sure what you mean by "a combination of both".

I usually extract and output the top 1K terms (by DF) and graph them to see where the elbow occurs, and pick that as the cut-off; terms with DF > the elbow go into the stopwords bucket.

Which has issues caused by any use of a step function, but in practice this typically hasn't been a significant problem.

One advantage is that often the set of stopwords changes with domains - e.g. when processing online ads, words like "free", "trial" and "download" are high DF terms.

And it's amazing how often (across multiple languages and domains) the DF limit winds up being 20%.

-- Ken

> My data set is one day's
> worth of news articles gathered from 1000 online news outlets. It's
> probably similar to the reuters data set, but with a little more
> noise. I used Boilerpipe for article extraction.
> 
> I spent a good while Googling around to build the largest (English)
> stop word-list I could. I'll paste it below for anyone who's
> interested and would like to save themselves an hour of Googling and
> collating.
> 
> -------------------------------------------------------------
> 
> 'll,'tis,'twas,a,a's,aan,able,about,above,abroad,abst,accordance,according,accordingly,across,act,actually,ad,added,adj,af,affected,affecting,affects,after,afterwards,again,against,ago,ah,ahead,ain't,al,all,alle,alles,allow,allows,almost,alone,along,alongside,already,als,also,alt,although,altijd,always,am,amid,amidst,among,amongst,amoungst,amount,an,and,anden,andere,announce,another,any,anybody,anyhow,anymore,anyone,anything,anyway,anyways,anywhere,apart,apparently,appear,appreciate,appropriate,approximately,are,area,areas,aren,aren't,arent,arise,around,as,aside,ask,asked,asking,asks,associated,at,auth,available,away,awfully,b,back,backed,backing,backs,backward,backwards,be,became,because,become,becomes,becoming,been,before,beforehand,began,begin,beginning,beginnings,begins,behind,being,beings,believe,below,ben,beside,besides,best,better,between,beyond,big,bij,bill,biol,blev,blive,bliver,both,bottom,brief,briefly,but,by,c,c'mon,c's,ca,call,came,can,can't,cannot,cant,caption,case,cases,cause,causes,certain,certainly,changes,clear,clearly,co,co.,com,come,comes,computer,con,concerning,consequently,consider,considering,contain,containing,contains,corresponding,could,could've,couldn't,couldnt,course,cry,currently,d,da,daar,dan,dare,daren't,dat,date,de,dear,definitely,dem,den,denne,der,deres,describe,described,despite,det,detail,dette,deze,did,didn't,die,differ,different,differently,dig,din,directly,disse,dit,do,doch,doen,does,doesn't,dog,doing,don,don't,done,door,down,downed,downing,downs,downwards,du,due,during,dus,e,each,early,ed,edu,een,eens,effect,efter,eg,eight,eighty,either,eleven,eller,else,elsewhere,empty,en,end,ended,ending,ends,enough,entirely,er,especially,et,et-al,etc,even,evenly,ever,evermore,every,everybody,everyone,everything,everywhere,ex,exactly,example,except,f,face,faces,fact,facts,fairly,far,farther,felt,few,fewer,ff,fifteen,fifth,fify,fill,find,finds,fire,first,five,fix,followed,following,follows,for,forever,former,formerly,forth,forty,forward,found,four,fra,from,front,full,fully,further,furthered,furthering,furthermore,furthers,g,gave,ge,geen,general,generally,get,gets,getting,geweest,give,given,gives,giving,go,goes,going,gone,good,goods,got,gotten,great,greater,greatest,greeting,greetings,group,grouped,grouping,groups,h,haar,had,hadn't,half,ham,han,hans,happen,happens,har,hardly,has,hasn't,hasnt,havde,have,haven't,having,he,he'd,he'll,he's,heb,hebben,hed,heeft,hello,help,hem,hence,hende,hendes,her,here,here's,hereafter,hereby,herein,heres,hereupon,hers,herself,hes,het,hi,hid,hier,high,higher,highest,hij,him,himself,his,hither,hoe,home,hopeful,hopefully,hos,how,how'd,how'll,how's,howbeit,however,hun,hundred,hvad,hvis,hvor,i,i'd,i'll,i'm,i've,id,ie,iemand,iets,if,ignored,ik,ikke,im,immediate,immediately,importance,important,in,inasmuch,inc,inc.,ind,indeed,index,indicate,indicated,indicates,information,inner,inside,insofar,instead,interest,interested,interesting,interests,into,invention,inward,is,isn't,it,it'd,it'll,it's,itd,its,itself,j,ja,je,jeg,jer,jo,just,k,kan,keep,keeps,kept,kg,kind,km,knew,know,knowing,known,knows,kon,kunne,kunnen,l,large,largely,last,lastly,late,lately,later,latest,latter,latterly,least,less,lest,let,let's,lets,like,liked,likely,likewise,line,little,long,longer,longest,look,looking,looks,low,lower,ltd,m,maar,made,mainly,make,makes,making,man,mange,many,may,maybe,mayn't,me,mean,means,meantime,meanwhile,med,meer,meget,member,members,men,merely,met,mg,mig,might,might've,mightn't,mij,mijn,mill,million,millions,min,mine,minus,miss,mit,ml,mod,moet,more,moreover,most,mostly,move,mr,mrs,much,must,must've,mustn't,my,myself,n,na,naar,name,namely,nay,nd,near,nearly,necessarily,necessary,ned,need,needed,needing,needn't,needs,neither,never,neverf,neverless,nevertheless,new,newer,newest,next,niet,niets,nine,ninety,no,no-one,nobody,nog,noget,nogle,non,none,nonetheless,noone,nor,normally,nos,not,noted,nothing,notwithstanding,novel,now,nowhere,nu,number,numbers,o,obtain,obtained,obviously,of,off,often,og,ogs,oh,ok,okay,old,older,oldest,om,omdat,omitted,on,once,onder,one,one's,ones,only,ons,onto,ook,op,open,opened,opening,opens,opposite,or,ord,order,ordered,ordering,orders,os,other,others,otherwise,ought,oughtn't,our,ours,ourselves,out,outside,over,overall,owing,own,p,page,pages,part,parted,particular,particularly,parting,parts,past,per,perhaps,place,placed,places,please,plus,point,pointed,pointing,points,poorly,possible,possibly,potentially,pp,predominantly,present,presented,presenting,presents,presumably,presume,presumed,previously,primarily,primary,probable,probably,problem,problems,prompt,promptly,proud,provide,provided,provides,put,puts,q,que,quick,quickly,quite,qv,r,ran,rather,rd,re,readily,really,reasonable,reasonably,recent,recently,reeds,ref,refs,regard,regarding,regardless,regards,relate,related,relative,relatively,respective,respectively,result,resulted,resulting,results,right,room,rooms,round,run,s,said,same,saw,say,saying,says,sec,second,secondly,seconds,see,seeing,seem,seemed,seeming,seems,seen,sees,self,selv,selves,sensible,sent,serious,seriously,seven,several,shall,shan't,she,she'd,she'll,she's,shed,shes,should,should've,shouldn't,show,showed,showing,shown,showns,shows,side,sides,sig,significant,significantly,similar,similarly,sin,since,sincere,sine,sit,six,sixty,skal,skulle,slightly,small,smaller,smallest,so,som,some,somebody,someday,somehow,someone,somethan,something,sometime,sometimes,somewhat,somewhere,soon,sorry,specifically,specified,specify,specifying,state,states,still,stop,strongl,strongly,sub,substantial,substantially,successfully,such,sufficient,sufficiently,suggest,suggested,suggests,sup,sure,system,t,t's,take,taken,taking,te,tegen,tell,tells,ten,tends,th,than,thank,thanks,thanx,that,that'll,that's,that've,thats,the,their,theirs,them,themselves,then,thence,there,there'd,there'll,there're,there's,there've,thereafter,thereby,therefore,therein,theres,thereupon,these,they,they'd,they'll,they're,they've,thi,thick,thin,thing,things,think,thinks,third,thirty,this,thorough,thoroughly,those,though,thought,thoughts,three,through,throughout,thru,thus,til,till,tis,to,toch,today,toen,together,too,took,top,tot,toward,towards,tried,tries,truly,try,trying,turn,turned,turning,turns,twas,twelve,twenty,twice,two,u,ud,uit,un,under,underneath,undoing,unfortunately,unless,unlike,unlikely,until,unto,up,upon,upwards,us,use,used,useful,uses,using,usually,uucp,uw,v,value,van,var,various,veel,versus,very,vi,via,vil,ville,viz,voor,vor,vs,være,været,w,want,wanted,wanting,wants,waren,was,wasn't,wat,way,ways,we,we'd,we'll,we're,we've,welcome,well,wells,went,werd,were,weren't,wezen,what,what'd,what'll,what's,what've,whatever,when,when'd,when'll,when's,whence,whenever,where,where'd,where'll,where's,whereafter,whereas,whereby,wherein,whereupon,wherever,whether,which,whichever,while,whilst,whither,who,who'd,who'll,who's,whoever,whole,whom,whomever,whose,why,why'd,why'll,why's,wie,wil,will,willing,wish,with,within,without,won't,wonder,worden,wordt,work,worked,working,works,would,would've,wouldn't,www,x,y,year,years,yes,yet,you,you'd,you'll,you're,you've,young,younger,youngest,your,yours,yourself,yourselves,z,zal,ze,zelf,zero,zich,zij,zijn,zo,zonder,zou

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

Re: stop word-lists VS maxDFPercent

[Ted Dunning <te...@gmail.com>]

I usually recommend using a custom stop list based on your own corpus.  That tends to work much better than general ones. 

I like using the doc frequency limit as well in case something goes strange on me. 

Sent from my iPhone

> On May 12, 2014, at 6:24, David Noel <da...@gmail.com> wrote:
> 
> What's everyone's opinion on using large stop word lists vs a very
> small value for maxDFPercent (like 30)? I'm playing around with both
> and am having trouble deciding whether one is better than the other,
> or if I should use a combination of both. My data set is one day's
> worth of news articles gathered from 1000 online news outlets. It's
> probably similar to the reuters data set, but with a little more
> noise. I used Boilerpipe for article extraction.
> 
> I spent a good while Googling around to build the largest (English)
> stop word-list I could. I'll paste it below for anyone who's
> interested and would like to save themselves an hour of Googling and
> collating.