You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Nicola Piazzi <Ni...@gruppocomet.it> on 2016/09/28 13:02:17 UTC
regular expression needed
Usually we receive spam having subjects like these examples in order of time :
Subject From To
FedEx Shipment 702193383647 Notification joe@company1.com sam@mycompany.it
FedEx Shipment 722566383641 Notification adam@other.com adam@mycompany.it
FedEx Shipment 734563383644 Notification info@company1.com lorel@mycompany.it
A package for you jim bill@cocacola.com jim@mycompany.it
A package for you sue joe@buster.com sue@mycompany.it
These come from viruses that infect different pcs in the word that send same spam
I want to write a plugin that test each email giving penality to these mails
Detection routine
A mail arrive
Subject is : FedEx Shipment 702193383647 Notification
I search in maillog table for a regex that MATCH FedEx Shipment 702193383647 Notification ALSO IN FedEx Shipment 722566383641 Notification AND IN FedEx Shipment 734563383644 Notification
If it match I verify that FROM DOMAIN IS DIFFERENT
And then I verify that TO ADDRESS IS DIFFERENT
Now I need a regex sintax to put all extracted words of PHRASE FedEx Shipment 734563383644 Notification and match if it found at least 3 of 4 words
Someone can help ?
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Re: regular expression needed
Posted by David Jones <dj...@ena.com>.
>A mail arrive
>Subject is : FedEx Shipment 702193383647 Notification
>I search in maillog table for a regex that MATCH FedEx Shipment 702193383647 Notification ALSO IN FedEx >Shipment 722566383641 Notification AND IN FedEx Shipment 734563383644 Notification
>If it match I verify that FROM DOMAIN IS DIFFERENT
>And then I verify that TO ADDRESS IS DIFFERENT
>Now I need a regex sintax to put all extracted words of PHRASE FedEx Shipment 734563383644 Notification and >match if it found at least 3 of 4 words
>Someone can help ?
This is a very manual way to block email that will never end. The
spammers will always be ahead of you. You should consider a
stronger reputation-based blocking approach.
1. Setup good RBLs (Postfix postscreen weighting). This is the most
important thing that will reject the majority of spam so SA only has
to process a very small percentage of email.
Note you have to whitelist major ISPs that unfortunately must be
bypassed from RBL checking since they are so large. Use postwhite
for this.
2. Use "whitelist_auth *@fedex.com" with SHORTCIRCUIT'ing
to allow real email from Fedex to get through. You can also
enable SHORTCIRCUIT'ing on other rule hits like:
shortcircuit RCVD_IN_MSPIKE_H5 on
shortcircuit RCVD_IN_RP_CERTIFIED on
shortcircuit RCVD_IN_RP_SAFE on
shortcircuit RCVD_IN_DNSWL_HI on
shortcircuit RCVD_IN_IADB_LISTED on
3. Train the Bayes database as spam for these forged emails.
4. If possible, purchase a feed (low cost) of the Invaluement RBL
and URIBLs and integrate it into SA with high scores since they are
very reliable.
5. Increase the score for RDNS_NONE.
6. Build a good list of whitelist_auth and whitelist_rcvd_from for
commonly spoofed senders like fedex.com, facebookmail.com,
etc. to extend the default list with ebay.com and paypal.com.
Hope this helps,
Dave
>Nicola Piazzi
>CED - Sistemi
Re: R: R: regular expression needed
Posted by John Hardin <jh...@impsec.org>.
On Wed, 28 Sep 2016, Bowie Bailey wrote:
> I don't think you could do a simple regex match for what you want. As I said
> previously, this would require a plugin both to build the custom regex(s) (or
> DB query) and to search for the previous emails. You would want to keep the
> prior email information in a database of some sort since doing a search of a
> large text file for every incoming email would probably be too slow.
That sounds like two-word Bayes to me... :)
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The reason it took so long to get Bin Laden is that it took the
SEALs five years to swim that far into the desert. -- anon
-----------------------------------------------------------------------
281 days since the first successful real return to launch site (SpaceX)
Re: R: R: R: regular expression needed
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 28 Sep 2016, at 11:20, Nicola Piazzi wrote:
> But I need the regex syntax to search at least 3 words that match of 4
> words given
This can be done with a SA meta rule:
header __FOO_IN_SUBJ Subject =~ /foo/i
header __BAR_IN_SUBJ Subject =~ /bar/i
header __BAZ_IN_SUBJ Subject =~ /baz/i
header __BIGNUM_IN_SUBJ Subject =~ /\d{8}/
meta __FOO_IN_SUBJ + __BAR_IN_SUBJ + __BAZ_IN_SUBJ + __BIGNUM_IN_SUBJ >
2
I see 34 such additive rules in the stock ruleset.
You could instead use a syntax like this in one rule:
header 3_OF_4_BAD_RES Subject =~ /(\b(foo|bar|baz|\d{8,})\b.*){3}/i
However, there is a small risk of a pathologically long Subject header
causing that to be very slow. If you want to do something like that in a
body rule you'd have a much larger risk of such slowness (especially if
you scan large messages) so you might prefer something like this:
body 3_OF_4_BAD_RES /(\b(foo|bar|baz|\d{8,})\b.{,50}){3}/i
The single-RE form is easier to translate into the context of something
other than SA rules (e.g. a Plugin) but it is less easily adapted to
other cases.
Re: regular expression needed
Posted by Bowie Bailey <Bo...@BUC.com>.
I am not familiar with MySQL regular expressions. The regex I provided
uses Perl syntax. It doesn't use lookarounds.
I can think of two things offhand that might be an issue.
1) Maybe it doesn't like the non-capturing groups. (?: .... )
Try it without the "?:" on the two groups:
\b((FedEx|Shipment|702193383246|Notification)\b.*?){3}
2) Maybe it doesn't like the non-greedy repetition. *?
Try this:
\b((FedEx|Shipment|702193383246|Notification)\b.*){3}
If neither of those work, try asking on a MySQL mailing list where they
should be more familiar with the syntax.
Keep in mind that this single regex will not do everything you want. It
is perfectly capable of matching the string "FedEx FedEx FedEx". You
need additional processing to make sure you are matching on 3 unique
strings and not just duplicates of one or two.
You may also want to make the regex case-insensitive. In Perl, this is
done with the "i" modifier ( /regex/i ), I have no idea how to do it
with MySQL.
Bowie
On 9/28/2016 12:28 PM, Nicola Piazzi wrote:
>
> This is what i need Bowie
>
> The query must be
>
> select from_address, from_domain, to_address, subject from maillog
> where subject REGEXP
> '\b(?:(?:FedEx|Shipment|702193383246|Notification)\b.*?){3}';
>
> But unfortunately mysql give error
>
> ERROR 1139 (42000): Got error 'repetition-operator operand invalid'
> from regexp
>
> MySQL regular expressions don't have lookarounds
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 17:46
> *A:* users@spamassassin.apache.org
> *Oggetto:* Re: R: R: R: regular expression needed
>
> I don't know of a way to do that with a simple regex. But since you
> are writing a plugin, you could do it by parsing the output of a regex
> search.
>
> 1) Create a regex which will match on any combination of 3 of the
> words. This will let you pull all of the possible matches from
> previous emails.
> Something like this: /\b(?:(?:word1|word2|word3|word4)\b.*?){3}/
>
> 2) For each of the lines found by the previous regex, run another
> regex that captures all matched words.
> /\b(word1|word2|word3|word4)\b/g (note the global modifier to catch
> all matches)
>
> 3) Take a look at the results for each line and see if the regex
> matched at least 3 unique words.
>
> I'm quite sure that this is not the most efficient method, but it
> should work.
>
> Bowie
>
> On 9/28/2016 11:20 AM, Nicola Piazzi wrote:
>
> Obviously i intended to write a plugin that search the db
>
> But I need the regex syntax to search at least 3 words that match
> of 4 words given
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 17:17
> *A:* Nicola Piazzi <Ni...@gruppocomet.it>
> <ma...@gruppocomet.it>; Spamassassin List
> <us...@spamassassin.apache.org> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: R: R: regular expression needed
>
> Please keep list emails on the list.
>
> I don't think you could do a simple regex match for what you
> want. As I said previously, this would require a plugin both to
> build the custom regex(s) (or DB query) and to search for the
> previous emails. You would want to keep the prior email
> information in a database of some sort since doing a search of a
> large text file for every incoming email would probably be too slow.
>
> Bowie
>
> On 9/28/2016 10:05 AM, Nicola Piazzi wrote:
>
> Flux :
>
> I receive an email with subject “Federal Express Important
> invoice number 20”
>
> Plugin search a regex in maillog database for 10 days ago
> mails and this regex search match 1 or more lines
>
> So we know that similar mails received in the past
>
> But it is normal to receive similar text but not so normal to
> receive same subject from different addresses directed to
> different internal users
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 16:01
> *A:* users@spamassassin.apache.org
> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: R: regular expression needed
>
> I'm still not clear on exactly what you are trying to do, but
> in order to test anything against previous messages, you will
> need a custom SA plugin and some sort of database to store the
> information about previous emails. That is beyond my area of
> expertise.
>
> If you just need a regex to match something, I'd be happy to
> help, but I would need a more explicit description of what you
> are trying to match.
>
> Bowie
>
> On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
>
> Bowie, your ia a manual way, it works but is not automated
>
> Automation is a plugin that check similar words in oldest
> messages (for example 3 of 4 words match)
>
> Then plugin check if sender domain is different and
> recipient is different
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 15:26
> *A:* users@spamassassin.apache.org
> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: regular expression needed
>
> On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
>
>
>
>
> Usually we receive spam having subjects like these
> examples in order of time :
>
>
>
>
>
> Subject From
> To
>
> FedEx Shipment 702193383647 Notification
> joe@company1.com <ma...@company1.com>
> sam@mycompany.it <ma...@mycompany.it>
>
> FedEx Shipment 722566383641 Notification
> adam@other.com <ma...@other.com>
> adam@mycompany.it <ma...@mycompany.it>
>
> FedEx Shipment 734563383644 Notification
> info@company1.com <ma...@company1.com>
> lorel@mycompany.it <ma...@mycompany.it>
>
> A package for you jim bill@cocacola.com
> <ma...@cocacola.com> jim@mycompany.it
> <ma...@mycompany.it>
>
> A package for you sue joe@buster.com
> <ma...@buster.com> sue@mycompany.it
> <ma...@mycompany.it>
>
> These come from viruses that infect different pcs in
> the word that send same spam
>
> I want to write a plugin that test each email giving
> penality to these mails
>
> Detection routine
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx
> Shipment 702193383647 Notification ALSO IN FedEx
> Shipment 722566383641 Notification AND IN FedEx
> Shipment 734563383644 Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
> Now I need a regex sintax to put all extracted words
> of PHRASE FedEx Shipment 734563383644 Notification and
> match if it found at least 3 of 4 words
>
> Someone can help ?
>
>
> I don't follow exactly what you are trying to do in the
> description above, but for that problem, I would start
> with something like this:
>
> header __FEDEX_ADDR From:addr /\@fedex\.com/
> header __FEDEX_SUBJ Subject /FedEx Shipment/
> meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
> score FEDEX_SPAM 2.0
>
> (Off the top of my head and completely untested. Adjust
> score as required.)
>
> This will hit any email with "FedEx Shipment" in the
> subject that doesn't come from fedex.com. Note that it
> will also hit on any legitimate FedEx emails that have
> been forwarded. You could minimize this by constraining
> the subject match to be at the beginning of the line
> (/^Fedex Shipment/). This may or may not have an effect
> on spam detection. You could also do a test for non-FedEx
> urls in the body rather than looking at the sender.
>
> You could use a simple subject line test for the "A
> package for you" emails, unless you know of a valid
> delivery service that uses that phrase.
>
> --
> Bowie
>
R: R: R: R: regular expression needed
Posted by Nicola Piazzi <Ni...@gruppocomet.it>.
This is what i need Bowie
The query must be
select from_address, from_domain, to_address, subject from maillog where subject REGEXP '\b(?:(?:FedEx|Shipment|702193383246|Notification)\b.*?){3}';
But unfortunately mysql give error
ERROR 1139 (42000): Got error 'repetition-operator operand invalid' from regexp
MySQL regular expressions don't have lookarounds
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 17:46
A: users@spamassassin.apache.org
Oggetto: Re: R: R: R: regular expression needed
I don't know of a way to do that with a simple regex. But since you are writing a plugin, you could do it by parsing the output of a regex search.
1) Create a regex which will match on any combination of 3 of the words. This will let you pull all of the possible matches from previous emails.
Something like this: /\b(?:(?:word1|word2|word3|word4)\b.*?){3}/
2) For each of the lines found by the previous regex, run another regex that captures all matched words.
/\b(word1|word2|word3|word4)\b/g (note the global modifier to catch all matches)
3) Take a look at the results for each line and see if the regex matched at least 3 unique words.
I'm quite sure that this is not the most efficient method, but it should work.
Bowie
On 9/28/2016 11:20 AM, Nicola Piazzi wrote:
Obviously i intended to write a plugin that search the db
But I need the regex syntax to search at least 3 words that match of 4 words given
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 17:17
A: Nicola Piazzi <Ni...@gruppocomet.it>; Spamassassin List <us...@spamassassin.apache.org>
Oggetto: Re: R: R: regular expression needed
Please keep list emails on the list.
I don't think you could do a simple regex match for what you want. As I said previously, this would require a plugin both to build the custom regex(s) (or DB query) and to search for the previous emails. You would want to keep the prior email information in a database of some sort since doing a search of a large text file for every incoming email would probably be too slow.
Bowie
On 9/28/2016 10:05 AM, Nicola Piazzi wrote:
Flux :
I receive an email with subject "Federal Express Important invoice number 20"
Plugin search a regex in maillog database for 10 days ago mails and this regex search match 1 or more lines
So we know that similar mails received in the past
But it is normal to receive similar text but not so normal to receive same subject from different addresses directed to different internal users
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 16:01
A: users@spamassassin.apache.org<ma...@spamassassin.apache.org>
Oggetto: Re: R: regular expression needed
I'm still not clear on exactly what you are trying to do, but in order to test anything against previous messages, you will need a custom SA plugin and some sort of database to store the information about previous emails. That is beyond my area of expertise.
If you just need a regex to match something, I'd be happy to help, but I would need a more explicit description of what you are trying to match.
Bowie
On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
Bowie, your ia a manual way, it works but is not automated
Automation is a plugin that check similar words in oldest messages (for example 3 of 4 words match)
Then plugin check if sender domain is different and recipient is different
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 15:26
A: users@spamassassin.apache.org<ma...@spamassassin.apache.org>
Oggetto: Re: regular expression needed
On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
Usually we receive spam having subjects like these examples in order of time :
Subject From To
FedEx Shipment 702193383647 Notification joe@company1.com<ma...@company1.com> sam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 722566383641 Notification adam@other.com<ma...@other.com> adam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 734563383644 Notification info@company1.com<ma...@company1.com> lorel@mycompany.it<ma...@mycompany.it>
A package for you jim bill@cocacola.com<ma...@cocacola.com> jim@mycompany.it<ma...@mycompany.it>
A package for you sue joe@buster.com<ma...@buster.com> sue@mycompany.it<ma...@mycompany.it>
These come from viruses that infect different pcs in the word that send same spam
I want to write a plugin that test each email giving penality to these mails
Detection routine
A mail arrive
Subject is : FedEx Shipment 702193383647 Notification
I search in maillog table for a regex that MATCH FedEx Shipment 702193383647 Notification ALSO IN FedEx Shipment 722566383641 Notification AND IN FedEx Shipment 734563383644 Notification
If it match I verify that FROM DOMAIN IS DIFFERENT
And then I verify that TO ADDRESS IS DIFFERENT
Now I need a regex sintax to put all extracted words of PHRASE FedEx Shipment 734563383644 Notification and match if it found at least 3 of 4 words
Someone can help ?
I don't follow exactly what you are trying to do in the description above, but for that problem, I would start with something like this:
header __FEDEX_ADDR From:addr /\@fedex\.com/
header __FEDEX_SUBJ Subject /FedEx Shipment/
meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
score FEDEX_SPAM 2.0
(Off the top of my head and completely untested. Adjust score as required.)
This will hit any email with "FedEx Shipment" in the subject that doesn't come from fedex.com. Note that it will also hit on any legitimate FedEx emails that have been forwarded. You could minimize this by constraining the subject match to be at the beginning of the line (/^Fedex Shipment/). This may or may not have an effect on spam detection. You could also do a test for non-FedEx urls in the body rather than looking at the sender.
You could use a simple subject line test for the "A package for you" emails, unless you know of a valid delivery service that uses that phrase.
--
Bowie
Re: R: R: R: regular expression needed
Posted by Bowie Bailey <Bo...@BUC.com>.
I don't know of a way to do that with a simple regex. But since you are
writing a plugin, you could do it by parsing the output of a regex search.
1) Create a regex which will match on any combination of 3 of the
words. This will let you pull all of the possible matches from previous
emails.
Something like this: /\b(?:(?:word1|word2|word3|word4)\b.*?){3}/
2) For each of the lines found by the previous regex, run another regex
that captures all matched words.
/\b(word1|word2|word3|word4)\b/g (note the global modifier to catch
all matches)
3) Take a look at the results for each line and see if the regex matched
at least 3 unique words.
I'm quite sure that this is not the most efficient method, but it should
work.
Bowie
On 9/28/2016 11:20 AM, Nicola Piazzi wrote:
>
> Obviously i intended to write a plugin that search the db
>
> But I need the regex syntax to search at least 3 words that match of 4
> words given
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 17:17
> *A:* Nicola Piazzi <Ni...@gruppocomet.it>; Spamassassin List
> <us...@spamassassin.apache.org>
> *Oggetto:* Re: R: R: regular expression needed
>
> Please keep list emails on the list.
>
> I don't think you could do a simple regex match for what you want. As
> I said previously, this would require a plugin both to build the
> custom regex(s) (or DB query) and to search for the previous emails.
> You would want to keep the prior email information in a database of
> some sort since doing a search of a large text file for every incoming
> email would probably be too slow.
>
> Bowie
>
> On 9/28/2016 10:05 AM, Nicola Piazzi wrote:
>
> Flux :
>
> I receive an email with subject “Federal Express Important invoice
> number 20”
>
> Plugin search a regex in maillog database for 10 days ago mails
> and this regex search match 1 or more lines
>
> So we know that similar mails received in the past
>
> But it is normal to receive similar text but not so normal to
> receive same subject from different addresses directed to
> different internal users
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 16:01
> *A:* users@spamassassin.apache.org
> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: R: regular expression needed
>
> I'm still not clear on exactly what you are trying to do, but in
> order to test anything against previous messages, you will need a
> custom SA plugin and some sort of database to store the
> information about previous emails. That is beyond my area of
> expertise.
>
> If you just need a regex to match something, I'd be happy to help,
> but I would need a more explicit description of what you are
> trying to match.
>
> Bowie
>
> On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
>
> Bowie, your ia a manual way, it works but is not automated
>
> Automation is a plugin that check similar words in oldest
> messages (for example 3 of 4 words match)
>
> Then plugin check if sender domain is different and recipient
> is different
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 15:26
> *A:* users@spamassassin.apache.org
> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: regular expression needed
>
> On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
>
>
>
> Usually we receive spam having subjects like these
> examples in order of time :
>
>
>
>
> Subject From To
>
> FedEx Shipment 702193383647 Notification joe@company1.com
> <ma...@company1.com> sam@mycompany.it
> <ma...@mycompany.it>
>
> FedEx Shipment 722566383641 Notification adam@other.com
> <ma...@other.com> adam@mycompany.it
> <ma...@mycompany.it>
>
> FedEx Shipment 734563383644 Notification info@company1.com
> <ma...@company1.com> lorel@mycompany.it
> <ma...@mycompany.it>
>
> A package for you jim bill@cocacola.com
> <ma...@cocacola.com> jim@mycompany.it
> <ma...@mycompany.it>
>
> A package for you sue joe@buster.com
> <ma...@buster.com> sue@mycompany.it
> <ma...@mycompany.it>
>
> These come from viruses that infect different pcs in the
> word that send same spam
>
> I want to write a plugin that test each email giving
> penality to these mails
>
> Detection routine
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx
> Shipment 702193383647 Notification ALSO IN FedEx Shipment
> 722566383641 Notification AND IN FedEx Shipment
> 734563383644 Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
> Now I need a regex sintax to put all extracted words of
> PHRASE FedEx Shipment 734563383644 Notification and match
> if it found at least 3 of 4 words
>
> Someone can help ?
>
>
> I don't follow exactly what you are trying to do in the
> description above, but for that problem, I would start with
> something like this:
>
> header __FEDEX_ADDR From:addr /\@fedex\.com/
> header __FEDEX_SUBJ Subject /FedEx Shipment/
> meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
> score FEDEX_SPAM 2.0
>
> (Off the top of my head and completely untested. Adjust score
> as required.)
>
> This will hit any email with "FedEx Shipment" in the subject
> that doesn't come from fedex.com. Note that it will also hit
> on any legitimate FedEx emails that have been forwarded. You
> could minimize this by constraining the subject match to be at
> the beginning of the line (/^Fedex Shipment/). This may or
> may not have an effect on spam detection. You could also do a
> test for non-FedEx urls in the body rather than looking at the
> sender.
>
> You could use a simple subject line test for the "A package
> for you" emails, unless you know of a valid delivery service
> that uses that phrase.
>
> --
> Bowie
>
R: R: R: regular expression needed
Posted by Nicola Piazzi <Ni...@gruppocomet.it>.
Obviously i intended to write a plugin that search the db
But I need the regex syntax to search at least 3 words that match of 4 words given
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 17:17
A: Nicola Piazzi <Ni...@gruppocomet.it>; Spamassassin List <us...@spamassassin.apache.org>
Oggetto: Re: R: R: regular expression needed
Please keep list emails on the list.
I don't think you could do a simple regex match for what you want. As I said previously, this would require a plugin both to build the custom regex(s) (or DB query) and to search for the previous emails. You would want to keep the prior email information in a database of some sort since doing a search of a large text file for every incoming email would probably be too slow.
Bowie
On 9/28/2016 10:05 AM, Nicola Piazzi wrote:
Flux :
I receive an email with subject "Federal Express Important invoice number 20"
Plugin search a regex in maillog database for 10 days ago mails and this regex search match 1 or more lines
So we know that similar mails received in the past
But it is normal to receive similar text but not so normal to receive same subject from different addresses directed to different internal users
Nicola Piazzi
CED - Sistemi
COMET s.p.a.
Via Michelino, 105 - 40127 Bologna - Italia
Tel. +39 051.6079.293
Cell. +39 328.21.73.470
Web: www.gruppocomet.it<http://www.gruppocomet.it/>
[Descrizione: gc]
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 16:01
A: users@spamassassin.apache.org<ma...@spamassassin.apache.org>
Oggetto: Re: R: regular expression needed
I'm still not clear on exactly what you are trying to do, but in order to test anything against previous messages, you will need a custom SA plugin and some sort of database to store the information about previous emails. That is beyond my area of expertise.
If you just need a regex to match something, I'd be happy to help, but I would need a more explicit description of what you are trying to match.
Bowie
On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
Bowie, your ia a manual way, it works but is not automated
Automation is a plugin that check similar words in oldest messages (for example 3 of 4 words match)
Then plugin check if sender domain is different and recipient is different
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 15:26
A: users@spamassassin.apache.org<ma...@spamassassin.apache.org>
Oggetto: Re: regular expression needed
On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
Usually we receive spam having subjects like these examples in order of time :
Subject From To
FedEx Shipment 702193383647 Notification joe@company1.com<ma...@company1.com> sam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 722566383641 Notification adam@other.com<ma...@other.com> adam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 734563383644 Notification info@company1.com<ma...@company1.com> lorel@mycompany.it<ma...@mycompany.it>
A package for you jim bill@cocacola.com<ma...@cocacola.com> jim@mycompany.it<ma...@mycompany.it>
A package for you sue joe@buster.com<ma...@buster.com> sue@mycompany.it<ma...@mycompany.it>
These come from viruses that infect different pcs in the word that send same spam
I want to write a plugin that test each email giving penality to these mails
Detection routine
A mail arrive
Subject is : FedEx Shipment 702193383647 Notification
I search in maillog table for a regex that MATCH FedEx Shipment 702193383647 Notification ALSO IN FedEx Shipment 722566383641 Notification AND IN FedEx Shipment 734563383644 Notification
If it match I verify that FROM DOMAIN IS DIFFERENT
And then I verify that TO ADDRESS IS DIFFERENT
Now I need a regex sintax to put all extracted words of PHRASE FedEx Shipment 734563383644 Notification and match if it found at least 3 of 4 words
Someone can help ?
I don't follow exactly what you are trying to do in the description above, but for that problem, I would start with something like this:
header __FEDEX_ADDR From:addr /\@fedex\.com/
header __FEDEX_SUBJ Subject /FedEx Shipment/
meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
score FEDEX_SPAM 2.0
(Off the top of my head and completely untested. Adjust score as required.)
This will hit any email with "FedEx Shipment" in the subject that doesn't come from fedex.com. Note that it will also hit on any legitimate FedEx emails that have been forwarded. You could minimize this by constraining the subject match to be at the beginning of the line (/^Fedex Shipment/). This may or may not have an effect on spam detection. You could also do a test for non-FedEx urls in the body rather than looking at the sender.
You could use a simple subject line test for the "A package for you" emails, unless you know of a valid delivery service that uses that phrase.
--
Bowie
Re: R: R: regular expression needed
Posted by Bowie Bailey <Bo...@BUC.com>.
Please keep list emails on the list.
I don't think you could do a simple regex match for what you want. As I
said previously, this would require a plugin both to build the custom
regex(s) (or DB query) and to search for the previous emails. You would
want to keep the prior email information in a database of some sort
since doing a search of a large text file for every incoming email would
probably be too slow.
Bowie
On 9/28/2016 10:05 AM, Nicola Piazzi wrote:
>
> Flux :
>
> I receive an email with subject “Federal Express Important invoice
> number 20”
>
> Plugin search a regex in maillog database for 10 days ago mails and
> this regex search match 1 or more lines
>
> So we know that similar mails received in the past
>
> But it is normal to receive similar text but not so normal to receive
> same subject from different addresses directed to different internal users
>
> Nicola Piazzi
> CED - Sistemi
> COMET s.p.a.
> Via Michelino, 105 - 40127 Bologna – Italia
> Tel. +39 051.6079.293
> Cell. +39 328.21.73.470
> Web: www.gruppocomet.it <http://www.gruppocomet.it/>
> Descrizione: gc
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 16:01
> *A:* users@spamassassin.apache.org
> *Oggetto:* Re: R: regular expression needed
>
> I'm still not clear on exactly what you are trying to do, but in order
> to test anything against previous messages, you will need a custom SA
> plugin and some sort of database to store the information about
> previous emails. That is beyond my area of expertise.
>
> If you just need a regex to match something, I'd be happy to help, but
> I would need a more explicit description of what you are trying to match.
>
> Bowie
>
> On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
>
> Bowie, your ia a manual way, it works but is not automated
>
> Automation is a plugin that check similar words in oldest messages
> (for example 3 of 4 words match)
>
> Then plugin check if sender domain is different and recipient is
> different
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 15:26
> *A:* users@spamassassin.apache.org
> <ma...@spamassassin.apache.org>
> *Oggetto:* Re: regular expression needed
>
> On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
>
>
> Usually we receive spam having subjects like these examples in
> order of time :
>
>
>
> Subject From To
>
> FedEx Shipment 702193383647 Notification joe@company1.com
> <ma...@company1.com> sam@mycompany.it
> <ma...@mycompany.it>
>
> FedEx Shipment 722566383641 Notification adam@other.com
> <ma...@other.com> adam@mycompany.it
> <ma...@mycompany.it>
>
> FedEx Shipment 734563383644 Notification info@company1.com
> <ma...@company1.com> lorel@mycompany.it
> <ma...@mycompany.it>
>
> A package for you jim bill@cocacola.com
> <ma...@cocacola.com> jim@mycompany.it
> <ma...@mycompany.it>
>
> A package for you sue joe@buster.com <ma...@buster.com>
> sue@mycompany.it <ma...@mycompany.it>
>
> These come from viruses that infect different pcs in the word
> that send same spam
>
> I want to write a plugin that test each email giving penality
> to these mails
>
> Detection routine
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx
> Shipment 702193383647 Notification ALSO IN FedEx Shipment
> 722566383641 Notification AND IN FedEx Shipment 734563383644
> Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
> Now I need a regex sintax to put all extracted words of PHRASE
> FedEx Shipment 734563383644 Notification and match if it found
> at least 3 of 4 words
>
> Someone can help ?
>
>
> I don't follow exactly what you are trying to do in the
> description above, but for that problem, I would start with
> something like this:
>
> header __FEDEX_ADDR From:addr /\@fedex\.com/
> header __FEDEX_SUBJ Subject /FedEx Shipment/
> meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
> score FEDEX_SPAM 2.0
>
> (Off the top of my head and completely untested. Adjust score as
> required.)
>
> This will hit any email with "FedEx Shipment" in the subject that
> doesn't come from fedex.com. Note that it will also hit on any
> legitimate FedEx emails that have been forwarded. You could
> minimize this by constraining the subject match to be at the
> beginning of the line (/^Fedex Shipment/). This may or may not
> have an effect on spam detection. You could also do a test for
> non-FedEx urls in the body rather than looking at the sender.
>
> You could use a simple subject line test for the "A package for
> you" emails, unless you know of a valid delivery service that uses
> that phrase.
>
> --
> Bowie
>
Re: R: regular expression needed
Posted by Bowie Bailey <Bo...@BUC.com>.
I'm still not clear on exactly what you are trying to do, but in order
to test anything against previous messages, you will need a custom SA
plugin and some sort of database to store the information about previous
emails. That is beyond my area of expertise.
If you just need a regex to match something, I'd be happy to help, but I
would need a more explicit description of what you are trying to match.
Bowie
On 9/28/2016 9:29 AM, Nicola Piazzi wrote:
>
> Bowie, your ia a manual way, it works but is not automated
>
> Automation is a plugin that check similar words in oldest messages
> (for example 3 of 4 words match)
>
> Then plugin check if sender domain is different and recipient is different
>
> *Da:*Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
> *Inviato:* mercoledì 28 settembre 2016 15:26
> *A:* users@spamassassin.apache.org
> *Oggetto:* Re: regular expression needed
>
> On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
>
> Usually we receive spam having subjects like these examples in
> order of time :
>
>
> Subject From To
>
> FedEx Shipment 702193383647 Notification joe@company1.com
> <ma...@company1.com> sam@mycompany.it <ma...@mycompany.it>
>
> FedEx Shipment 722566383641 Notification adam@other.com
> <ma...@other.com> adam@mycompany.it <ma...@mycompany.it>
>
> FedEx Shipment 734563383644 Notification info@company1.com
> <ma...@company1.com> lorel@mycompany.it
> <ma...@mycompany.it>
>
> A package for you jim bill@cocacola.com <ma...@cocacola.com>
> jim@mycompany.it <ma...@mycompany.it>
>
> A package for you sue joe@buster.com <ma...@buster.com>
> sue@mycompany.it <ma...@mycompany.it>
>
> These come from viruses that infect different pcs in the word that
> send same spam
>
> I want to write a plugin that test each email giving penality to
> these mails
>
> Detection routine
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx Shipment
> 702193383647 Notification ALSO IN FedEx Shipment 722566383641
> Notification AND IN FedEx Shipment 734563383644 Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
> Now I need a regex sintax to put all extracted words of PHRASE
> FedEx Shipment 734563383644 Notification and match if it found at
> least 3 of 4 words
>
> Someone can help ?
>
>
> I don't follow exactly what you are trying to do in the description
> above, but for that problem, I would start with something like this:
>
> header __FEDEX_ADDR From:addr /\@fedex\.com/
> header __FEDEX_SUBJ Subject /FedEx Shipment/
> meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
> score FEDEX_SPAM 2.0
>
> (Off the top of my head and completely untested. Adjust score as
> required.)
>
> This will hit any email with "FedEx Shipment" in the subject that
> doesn't come from fedex.com. Note that it will also hit on any
> legitimate FedEx emails that have been forwarded. You could minimize
> this by constraining the subject match to be at the beginning of the
> line (/^Fedex Shipment/). This may or may not have an effect on spam
> detection. You could also do a test for non-FedEx urls in the body
> rather than looking at the sender.
>
> You could use a simple subject line test for the "A package for you"
> emails, unless you know of a valid delivery service that uses that phrase.
>
> --
> Bowie
>
Re: R: regular expression needed
Posted by Jesse Norell <je...@kci.net>.
On Wed, 2016-09-28 at 13:29 +0000, Nicola Piazzi wrote:
> a plugin that check similar words in oldest messages (for example 3 of
> 4 words match)
>
> Then plugin check if sender domain is different and recipient is
> different
<snip>
> Detection routine
>
>
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx
> Shipment 702193383647 Notification ALSO IN FedEx Shipment
> 722566383641 Notification AND IN FedEx Shipment 734563383644
> Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
>
>
> Now I need a regex sintax to put all extracted words of PHRASE
> FedEx Shipment 734563383644 Notification and match if it found
> at least 3 of 4 words
I'm also not clear on exactly what you're intending, but this certainly
sounds reminiscent of Marc Perkel's "evolution filter" (which I don't
know that anyone fully understands). What I've made out of the
discussion is it is token-based like bayes, using multi-word (and
partial-word/string?) tokens and adds some other data and metadata as
tokens (data from headers, eg. your from: and to: domains), and tosses
out results that aren't confident (nearly 100% ham or spam); it utilizes
Redis Sets for set logic/operations. If you are creating a plugin for
these phishing emails, it may be an avenue to pursue; it sounds like it
works quite well (when trained with a large ham/spam corpus).
--
Jesse Norell
Kentec Communications, Inc.
970-522-8107 - www.kci.net
R: regular expression needed
Posted by Nicola Piazzi <Ni...@gruppocomet.it>.
Bowie, your ia a manual way, it works but is not automated
Automation is a plugin that check similar words in oldest messages (for example 3 of 4 words match)
Then plugin check if sender domain is different and recipient is different
Da: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Inviato: mercoledì 28 settembre 2016 15:26
A: users@spamassassin.apache.org
Oggetto: Re: regular expression needed
On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
Usually we receive spam having subjects like these examples in order of time :
Subject From To
FedEx Shipment 702193383647 Notification joe@company1.com<ma...@company1.com> sam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 722566383641 Notification adam@other.com<ma...@other.com> adam@mycompany.it<ma...@mycompany.it>
FedEx Shipment 734563383644 Notification info@company1.com<ma...@company1.com> lorel@mycompany.it<ma...@mycompany.it>
A package for you jim bill@cocacola.com<ma...@cocacola.com> jim@mycompany.it<ma...@mycompany.it>
A package for you sue joe@buster.com<ma...@buster.com> sue@mycompany.it<ma...@mycompany.it>
These come from viruses that infect different pcs in the word that send same spam
I want to write a plugin that test each email giving penality to these mails
Detection routine
A mail arrive
Subject is : FedEx Shipment 702193383647 Notification
I search in maillog table for a regex that MATCH FedEx Shipment 702193383647 Notification ALSO IN FedEx Shipment 722566383641 Notification AND IN FedEx Shipment 734563383644 Notification
If it match I verify that FROM DOMAIN IS DIFFERENT
And then I verify that TO ADDRESS IS DIFFERENT
Now I need a regex sintax to put all extracted words of PHRASE FedEx Shipment 734563383644 Notification and match if it found at least 3 of 4 words
Someone can help ?
I don't follow exactly what you are trying to do in the description above, but for that problem, I would start with something like this:
header __FEDEX_ADDR From:addr /\@fedex\.com/
header __FEDEX_SUBJ Subject /FedEx Shipment/
meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
score FEDEX_SPAM 2.0
(Off the top of my head and completely untested. Adjust score as required.)
This will hit any email with "FedEx Shipment" in the subject that doesn't come from fedex.com. Note that it will also hit on any legitimate FedEx emails that have been forwarded. You could minimize this by constraining the subject match to be at the beginning of the line (/^Fedex Shipment/). This may or may not have an effect on spam detection. You could also do a test for non-FedEx urls in the body rather than looking at the sender.
You could use a simple subject line test for the "A package for you" emails, unless you know of a valid delivery service that uses that phrase.
--
Bowie
Re: regular expression needed
Posted by Bowie Bailey <Bo...@BUC.com>.
On 9/28/2016 9:02 AM, Nicola Piazzi wrote:
>
> Usually we receive spam having subjects like these examples in order
> of time :
>
> Subject From To
>
> FedEx Shipment 702193383647 Notification
> joe@company1.com sam@mycompany.it
>
> FedEx Shipment 722566383641 Notification
> adam@other.com adam@mycompany.it
>
> FedEx Shipment 734563383644 Notification
> info@company1.com lorel@mycompany.it
>
> A package for you jim bill@cocacola.com jim@mycompany.it
>
> A package for you sue joe@buster.com sue@mycompany.it
>
> These come from viruses that infect different pcs in the word that
> send same spam
>
> I want to write a plugin that test each email giving penality to these
> mails
>
> Detection routine
>
> A mail arrive
>
> Subject is : FedEx Shipment 702193383647 Notification
>
> I search in maillog table for a regex that MATCH FedEx Shipment
> 702193383647 Notification ALSO IN FedEx Shipment 722566383641
> Notification AND IN FedEx Shipment 734563383644 Notification
>
> If it match I verify that FROM DOMAIN IS DIFFERENT
> And then I verify that TO ADDRESS IS DIFFERENT
>
> Now I need a regex sintax to put all extracted words of PHRASE FedEx
> Shipment 734563383644 Notification and match if it found at least 3 of
> 4 words
>
> Someone can help ?
>
I don't follow exactly what you are trying to do in the description
above, but for that problem, I would start with something like this:
header __FEDEX_ADDR From:addr /\@fedex\.com/
header __FEDEX_SUBJ Subject /FedEx Shipment/
meta FEDEX_SPAM __FEDEX_SUBJ && ! __FEDEX_ADDR
score FEDEX_SPAM 2.0
(Off the top of my head and completely untested. Adjust score as required.)
This will hit any email with "FedEx Shipment" in the subject that
doesn't come from fedex.com. Note that it will also hit on any
legitimate FedEx emails that have been forwarded. You could minimize
this by constraining the subject match to be at the beginning of the
line (/^Fedex Shipment/). This may or may not have an effect on spam
detection. You could also do a test for non-FedEx urls in the body
rather than looking at the sender.
You could use a simple subject line test for the "A package for you"
emails, unless you know of a valid delivery service that uses that phrase.
--
Bowie