You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@spamassassin.apache.org on 2024/07/22 18:29:29 UTC
[Bug 8271] New: bayes_stopword_ru in 60_bayes_stopwords.cf misses some very common russian stopwords
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8271
Bug ID: 8271
Summary: bayes_stopword_ru in 60_bayes_stopwords.cf misses some
very common russian stopwords
Product: Spamassassin
Version: 4.0.1
Hardware: PC
OS: FreeBSD
Status: NEW
Severity: enhancement
Priority: P2
Component: Rules
Assignee: dev@spamassassin.apache.org
Reporter: azotov@geolink-group.com
Target Milestone: Undefined
The list of russian stopwords compiled into bayes_stopword_ru seems to be very
long but it misses some very common russian stopwords.
How to reproduce:
cat <<EOT >> stopwords.mbox
From test Mon Jul 22 20:42:19 2024
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
=D0=B5=D1=81=D0=BB=D0=B8 =D0=B8=D0=BB=D0=B8 =D0=BD=D0=B5
EOT
spamassassin -L -D bayes --mbox stopwords.mbox
Only the first stopword "если" is found:
dbg: bayes: skipped token '\x{D0}\x{B5}\x{D1}\x{81}\x{D0}\x{BB}\x{D0}\x{B8}'
because it's in stopword list for language 'ru'
Stopwords \x{D0}\x{B8}\x{D0}\x{BB}\x{D0}\x{B8} ("или" means "or" in english)
and \x{D0}\x{BD}\x{D0}\x{B5} ("не" means "not" in english) are missed.
The whole list of missed russian stopwords is "на|по|не|от|для|или|за|из|что".
It will be great to adjust 60_bayes_stopwords.cf by adding these stopwords to
bayes_stopword_ru rule.
P.S. Current version of bayes_stopword_ru from 60_bayes_stopwords.cf also
contains some low frequency russian words which cannot be considered as
stopwords. For example word "иногда" ("sometimes" in english) was found only in
7 of 8000 spam messages and in 50 from 40000 ham messages in my own corpus
(selected by hand).
Personally I use the following custom list of stopwords in my installation of
spamassassin:
bayes_stopword_ru
(?^:(на|по|не|от|для|или|за|Вас|из|что|если|будет|Вам|Если|мы|Здравствуйте|есть|это|можно|только|вас|нужно|без|его))
Each of these words had a Bayes score between 0.4 and 0.6 (for bayes db trained
on my corpus with no stopwords) and was found in at least 10% of messages from
my corpus.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8271] bayes_stopword_ru in 60_bayes_stopwords.cf misses some very common russian stopwords
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8271
Giovanni Bechis <gi...@paclan.it> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #3 from Giovanni Bechis <gi...@paclan.it> ---
Committed, thanks.
Sending rules/60_bayes_stopwords.cf
Transmitting file data .done
Committing transaction...
Committed revision 1919603.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8271] bayes_stopword_ru in 60_bayes_stopwords.cf misses some very common russian stopwords
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8271
azotov@geolink-group.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |azotov@geolink-group.com
--- Comment #2 from azotov@geolink-group.com ---
Yes, I understand that it is difficult to generate stopwords for a language you
don't know.
Some short russian stopwords (such as "не") could be skipped by your python
script because they have 2 characters accoring to Unicode-aware python but
actually consist of 4 bytes in Unicode encoding. The check "next if $len < 3;"
in sub _tokenize_line in Bayes.pm does not skip such tokens because their
length is 4 bytes. Such words (2 characters encoded with 4 bytes) should be
included in stopwords lists in any language despite looking too short at fist
sight.
You can use my russian stoplist for spamassassin rules.
It's human-readable form is:
bayes_stopword_ru
(?^:(на|по|не|от|для|или|за|Вас|из|что|если|будет|Вам|Если|мы|Здравствуйте|есть|это|можно|только|вас|нужно|без|его))
Optimized regexp:
bayes_stopword_ru
(?^:(?:\xd0(?:\x92\xd0\xb0(?:\xd0\xbc|\xd1\x81)|\x95\xd1\x81\xd0\xbb\xd0\xb8|\x97\xd0\xb4\xd1\x80\xd0\xb0\xd0\xb2\xd1\x81\xd1\x82\xd0\xb2\xd1\x83\xd0\xb9\xd1\x82\xd0\xb5|\xb1(?:\xd0\xb5\xd0\xb7|\xd1\x83\xd0\xb4\xd0\xb5\xd1\x82)|\xb2\xd0\xb0\xd1\x81|\xb4\xd0\xbb\xd1\x8f|\xb5(?:\xd0\xb3\xd0\xbe|\xd1\x81(?:\xd0\xbb\xd0\xb8|\xd1\x82\xd1\x8c))|\xb7\xd0\xb0|\xb8\xd0(?:\xb7|\xbb\xd0\xb8)|\xbc(?:\xd0\xbe\xd0\xb6\xd0\xbd\xd0\xbe|\xd1\x8b)|\xbd(?:\xd0(?:\xb0|\xb5)|\xd1\x83\xd0\xb6\xd0\xbd\xd0\xbe)|\xbe\xd1\x82|\xbf\xd0\xbe)|\xd1(?:\x82\xd0\xbe\xd0\xbb\xd1\x8c\xd0\xba\xd0\xbe|\x87\xd1\x82\xd0\xbe|\x8d\xd1\x82\xd0\xbe)))
I hope it's versatile enough. I tries to exclude all words which are specific
for our mail server message flow. I hope this list is better than "standard"
stoplists built from fiction books corpus and not from real messages.
The list consists of 24 words, most of words are also present in "standard"
stoplists (http://snowball.tartarus.org/algorithms/russian/stop.txt for
example). The only exceptions (my additions) are "Здравствуйте" (polite form of
"Hello") and "нужно" ("should" or "need" depending on context).
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8271] bayes_stopword_ru in 60_bayes_stopwords.cf misses some very common russian stopwords
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8271
Giovanni Bechis <gi...@paclan.it> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |giovanni@paclan.it
--- Comment #1 from Giovanni Bechis <gi...@paclan.it> ---
Since I am not a russian native speaker I've implemented this regexp using a
script that uses Python nltk module.
I would be glad if you can provide a better regexp that covers real common
russian words.
--
You are receiving this mail because:
You are the assignee for the bug.