You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Marcus Schopen <li...@localguru.de> on 2014/09/12 10:05:10 UTC

sa-learn from a remote imap folder

Hi,

spamassassin and imap (cyrus) are running on different boxes. What is
best practice to learn spam from a remote imap folder? Any good working
scripts?

I found these:

https://wiki.apache.org/spamassassin/RemoteImapFolder

https://gist.github.com/colinmollenhour/4127743

Ciao!
Marcus

Re: sa-learn from a remote imap folder

Posted by Kris Deugau <kd...@vianet.ca>.
Marcus Schopen wrote:
> Hi,
> 
> spamassassin and imap (cyrus) are running on different boxes. What is
> best practice to learn spam from a remote imap folder? Any good working
> scripts?

I've been using http://www.deepnet.cx/~kdeugau/spamtools/imap-learn
(derived from the script at http://www.dmzs.com/tools/files/spam.phtml)
in production here for quite a while.  (The timestamp on that particular
copy is from 2012;  pretty sure it hasn't changed for quite some time
before that.)

There are probably things it could do better or more efficiently, but it
works well enough I can't justify fiddling with it.

-kgd

Re: sa-learn from a remote imap folder

Posted by RW <rw...@googlemail.com>.
On Sat, 13 Sep 2014 13:54:13 -0500
Dave Pooser wrote:

> On 9/13/14 1:38 PM, "Bob Proulx" <bo...@proulx.com> wrote:
> 
> >If you are using Maildir format then there are no locks.  That makes
> >Maildir format the format of choice for use over NFS.
> 
> Excellent. I am in fact using Maildir, so I guess it's not just luck
> that I've had no locking issues.

Delivery into Maildir doesn't require a lock, but a well designed
IMAP server will still lock Maildir to avoid flag corruption. This is
unlikely to make a difference to sa-learn, unless you are trying to
do something exotic.

Re: sa-learn from a remote imap folder

Posted by Dave Pooser <da...@pooserville.com>.
On 9/13/14 1:38 PM, "Bob Proulx" <bo...@proulx.com> wrote:

>If you are using Maildir format then there are no locks.  That makes
>Maildir format the format of choice for use over NFS.

Excellent. I am in fact using Maildir, so I guess it's not just luck that
I've had no locking issues.

Thanks, Ian, for the warning and thanks, Bob, for setting my mind at ease.
;-) 
-- 
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com

Re: sa-learn from a remote imap folder

Posted by Bob Proulx <bo...@proulx.com>.
Dave Pooser wrote:
> >Dave> At $DAYJOB we export the spam folder (and a ham folder for FPs)
> >Dave> via NFS and mount them on the frontline SA servers for sa-learn.
> >
> >Doesn't that smell of locking issues?
> 
> To be honest, I'd just assumed that NFS wouldn't do any locking on a
> read-only export.

If you are using Maildir format then there are no locks.  That makes
Maildir format the format of choice for use over NFS.  Here is a
reference.

  http://www.postfix.org/NFS_README.html
  The maildir format uses one file per message and needs no file
  locking support in Postfix or in other mail software.

> ...Great, now I'm paranoid. Thanks, Ian. ;-) I guess I can go back and
> force the exports to ne NFS3 instead of NFS4 so I can explicitly state the
> nolocks option....

Paranoia is good.  But using Maildir format would also be okay too.

Bob

Re: sa-learn from a remote imap folder

Posted by Jari Fredriksson <ja...@iki.fi>.
13.09.2014, 08:57, Dave Pooser kirjoitti:
>> Dave> At $DAYJOB we export the spam folder (and a ham folder for FPs)
>> Dave> via NFS and mount them on the frontline SA servers for sa-learn.
>>
>> Doesn't that smell of locking issues?
> To be honest, I'd just assumed that NFS wouldn't do any locking on a
> read-only export. I haven't seen any issues yet, but we're talking small
> volume, and the frontline servers mount the directories periodically, run
> the sa-learn, and then unmount them again.
>
> ...Great, now I'm paranoid. Thanks, Ian. ;-) I guess I can go back and
> force the exports to ne NFS3 instead of NFS4 so I can explicitly state the
> nolocks option....

I have my Maildir on a NAS via NFS, but that is only a one user system
anyway.

-- 
jarif.bit

Re: sa-learn from a remote imap folder

Posted by Dave Pooser <da...@pooserville.com>.
>Dave> At $DAYJOB we export the spam folder (and a ham folder for FPs)
>Dave> via NFS and mount them on the frontline SA servers for sa-learn.
>
>Doesn't that smell of locking issues?

To be honest, I'd just assumed that NFS wouldn't do any locking on a
read-only export. I haven't seen any issues yet, but we're talking small
volume, and the frontline servers mount the directories periodically, run
the sa-learn, and then unmount them again.

...Great, now I'm paranoid. Thanks, Ian. ;-) I guess I can go back and
force the exports to ne NFS3 instead of NFS4 so I can explicitly state the
nolocks option....
-- 
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

Re: sa-learn from a remote imap folder

Posted by Ian Zimmerman <it...@buug.org>.
On Fri, 12 Sep 2014 07:45:22 -0500,
Dave Pooser <da...@pooserville.com> wrote:

Marcus> spamassassin and imap (cyrus) are running on different
Marcus> boxes. What is best practice to learn spam from a remote imap
Marcus> folder?

Dave> At $DAYJOB we export the spam folder (and a ham folder for FPs)
Dave> via NFS and mount them on the frontline SA servers for sa-learn.

Doesn't that smell of locking issues?

-- 
Please *no* private copies of mailing list or newsgroup messages.
Local Variables:
mode:claws-external
End:

Re: sa-learn from a remote imap folder

Posted by Dave Pooser <da...@pooserville.com>.
On 9/12/14 3:05 AM, "Marcus Schopen" <li...@localguru.de> wrote:

>spamassassin and imap (cyrus) are running on different boxes. What is
>best practice to learn spam from a remote imap folder?

At $DAYJOB we export the spam folder (and a ham folder for FPs) via NFS
and mount them on the frontline SA servers for sa-learn.
-- 
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

Re: sa-learn from a remote imap folder

Posted by Giles Coochey <gi...@coochey.net>.
On 12/09/2014 18:34, Rick Macdougall wrote:
> On 2014-09-12 1:24 PM, John Hardin wrote:
>> On Fri, 12 Sep 2014, Reindl Harald wrote:
>>
>>>
>>> Am 12.09.2014 um 15:26 schrieb Giles Coochey:
>>>> On 12/09/2014 13:47, Rick Macdougall wrote:
>>>>>
>>>>> I have used imap-sa-learn.pl for years.  Works great.
>>>>>
>>>>> Google imap-sa-learn.pl to get the perl source code.
>>>>>
>>>> Wouldn't mind using it, but don't think I can get it working as my
>>>> IMAP server requires SSL
>>>
>>> have you tried it?
>>>
>>> these days almost anything works with SSL
>>> because common used libraries
>>
>> Also: stunnel.
>>
>
> Or you know, set ssl => 1 and port => 993 where the program opens a 
> new connection.
>
> It's very straight forward.
>
Straightforward once the documentation is located and read...

So now it appears to connect, but doesn't appear to read my "Junk 
E-mail" folder correctly (perhaps due to handling of white space, or 
hyphen in the folder name?)...

-- 
Regards,

Giles Coochey, CCNP, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 8444 780677
+44 (0) 7584 634135
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net

Re: sa-learn from a remote imap folder

Posted by Rick Macdougall <ri...@ummm-beer.com>.
On 2014-09-12 1:24 PM, John Hardin wrote:
> On Fri, 12 Sep 2014, Reindl Harald wrote:
>
>>
>> Am 12.09.2014 um 15:26 schrieb Giles Coochey:
>>> On 12/09/2014 13:47, Rick Macdougall wrote:
>>>>
>>>> I have used imap-sa-learn.pl for years.  Works great.
>>>>
>>>> Google imap-sa-learn.pl to get the perl source code.
>>>>
>>> Wouldn't mind using it, but don't think I can get it working as my
>>> IMAP server requires SSL
>>
>> have you tried it?
>>
>> these days almost anything works with SSL
>> because common used libraries
>
> Also: stunnel.
>

Or you know, set ssl => 1 and port => 993 where the program opens a new 
connection.

It's very straight forward.

Regards,

Rick

Re: sa-learn from a remote imap folder

Posted by John Hardin <jh...@impsec.org>.
On Fri, 12 Sep 2014, Reindl Harald wrote:

>
> Am 12.09.2014 um 15:26 schrieb Giles Coochey:
>> On 12/09/2014 13:47, Rick Macdougall wrote:
>>>
>>> I have used imap-sa-learn.pl for years.  Works great.
>>>
>>> Google imap-sa-learn.pl to get the perl source code.
>>>
>> Wouldn't mind using it, but don't think I can get it working as my IMAP server requires SSL
>
> have you tried it?
>
> these days almost anything works with SSL
> because common used libraries

Also: stunnel.

-- 
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@impsec.org    FALaholic #11174     pgpk -a jhardin@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
   Maxim II: A Sergeant in motion outranks a Lieutenant who doesn't
   know what's going on.
   Maxim III: An ordnance technician at a dead run outranks everybody.
-----------------------------------------------------------------------
  5 days until the 227th anniversary of the signing of the U.S. Constitution

Re: sa-learn from a remote imap folder

Posted by Giles Coochey <gi...@coochey.net>.
On 12/09/2014 17:01, Reindl Harald wrote:
> Am 12.09.2014 um 17:48 schrieb Giles Coochey:
>> On 12/09/2014 14:30, Reindl Harald wrote:
>>>> Wouldn't mind using it, but don't think I can get it working as my IMAP server requires SSL
>>> have you tried it?
>>>
>>> these days almost anything works with SSL
>>> because common used libraries
>>>
>> It times out, as it tries to connect on port 143, server runs on port 993
> uhm Port 143 should offer and enforce STARTTLS independent
> of the script, just because some clients don't work with
> 993 and others not with STARTTLS
>
>
Yes, it's not what the server offers, it's about what is open on the 
firewall - which separates the smtp server (external DMZ), from the 
mailbox server (internal DMZ). The point is that the server does not 
support the certificate based SSL that is run by policy, STARTTLS is 
different altogether.

-- 
Regards,

Giles Coochey, CCNP, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 8444 780677
+44 (0) 7584 634135
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net

Re: sa-learn from a remote imap folder

Posted by Reindl Harald <h....@thelounge.net>.
Am 12.09.2014 um 17:48 schrieb Giles Coochey:
> On 12/09/2014 14:30, Reindl Harald wrote:
>>
>>> Wouldn't mind using it, but don't think I can get it working as my IMAP server requires SSL
>> have you tried it?
>>
>> these days almost anything works with SSL
>> because common used libraries
>>
> It times out, as it tries to connect on port 143, server runs on port 993

uhm Port 143 should offer and enforce STARTTLS independent
of the script, just because some clients don't work with
993 and others not with STARTTLS

Re: sa-learn from a remote imap folder

Posted by Giles Coochey <gi...@coochey.net>.
On 12/09/2014 14:30, Reindl Harald wrote:
>
>> Wouldn't mind using it, but don't think I can get it working as my IMAP server requires SSL
> have you tried it?
>
> these days almost anything works with SSL
> because common used libraries
>
It times out, as it tries to connect on port 143, server runs on port 993.

-- 
Regards,

Giles Coochey, CCNP, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 8444 780677
+44 (0) 7584 634135
http://www.netsecspec.co.uk
giles.coochey@netsecspec.co.uk



-- 
Regards,

Giles Coochey, CCNP, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 8444 780677
+44 (0) 7584 634135
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net

Re: sa-learn from a remote imap folder

Posted by Reindl Harald <h....@thelounge.net>.
Am 12.09.2014 um 15:26 schrieb Giles Coochey:
> On 12/09/2014 13:47, Rick Macdougall wrote:
>>
>> I have used imap-sa-learn.pl for years.  Works great.
>>
>> Google imap-sa-learn.pl to get the perl source code.
>>
> Wouldn't mind using it, but don't think I can get it working as my IMAP server requires SSL

have you tried it?

these days almost anything works with SSL
because common used libraries

Re: sa-learn from a remote imap folder

Posted by Giles Coochey <gi...@coochey.net>.
On 12/09/2014 13:47, Rick Macdougall wrote:
>
> Hi,
>
> I have used imap-sa-learn.pl for years.  Works great.
>
> Google imap-sa-learn.pl to get the perl source code.
>
Wouldn't mind using it, but don't think I can get it working as my IMAP 
server requires SSL

-- 
Regards,

Giles Coochey, CCNP, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 8444 780677
+44 (0) 7584 634135
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net

Re: sa-learn from a remote imap folder

Posted by Rick Macdougall <ri...@ummm-beer.com>.
>> Am Freitag, den 12.09.2014, 10:13 +0200 schrieb Axb:
>>> On 09/12/2014 10:05 AM, Marcus Schopen wrote:
>>>> Hi,
>>>>
>>>> spamassassin and imap (cyrus) are running on different boxes. What is
>>>> best practice to learn spam from a remote imap folder? Any good working
>>>> scripts?
>>>>

Hi,

I have used imap-sa-learn.pl for years.  Works great.

Google imap-sa-learn.pl to get the perl source code.

Regards,

Rick

RE: sa-learn from a remote imap folder

Posted by David Jones <dj...@ena.com>.
>________________________________________
>From: Marcus Schopen <li...@localguru.de>
>Sent: Friday, September 12, 2014 3:33 AM
>To: Axb
>Cc: users@spamassassin.apache.org
>Subject: Re: sa-learn from a remote imap folder

>Hi,

>Am Freitag, den 12.09.2014, 10:13 +0200 schrieb Axb:
>> On 09/12/2014 10:05 AM, Marcus Schopen wrote:
>> > Hi,
>> >
>> > spamassassin and imap (cyrus) are running on different boxes. What is
>> > best practice to learn spam from a remote imap folder? Any good working
>> > scripts?
>> >
>> > I found these:
>> >
>> > https://wiki.apache.org/spamassassin/RemoteImapFolder
>> >
>> > https://gist.github.com/colinmollenhour/4127743
>>
>> Imapsync or OfflineIMAP
>>
>> http://imapsync.lamiral.info/
>> http://offlineimap.org/


>On the spamassassin box there is no imap server running to have an imap
>to imap sync. Therefore I think imapsync can't handle this. offlineimap
>seems to be interesting, because it seems to "dump" the mails to a local
>folder, which could eaten by sa-learn.

>> there are tons more - Google "sync imap"

>Before start scripting mysself I was looking for a smart and stable
>script, which people use for long time without problems.

Check out fetchmail.  It combined with procmail can do just about anything
you need to do with email.  You may only need fetchmail in this case to
gather up the email for sa-learn.

>Ciao
>Marcus

Re: sa-learn from a remote imap folder

Posted by Marcus Schopen <li...@localguru.de>.
Hi,

Am Freitag, den 12.09.2014, 10:13 +0200 schrieb Axb:
> On 09/12/2014 10:05 AM, Marcus Schopen wrote:
> > Hi,
> >
> > spamassassin and imap (cyrus) are running on different boxes. What is
> > best practice to learn spam from a remote imap folder? Any good working
> > scripts?
> >
> > I found these:
> >
> > https://wiki.apache.org/spamassassin/RemoteImapFolder
> >
> > https://gist.github.com/colinmollenhour/4127743
> 
> Imapsync or OfflineIMAP
> 
> http://imapsync.lamiral.info/
> http://offlineimap.org/


On the spamassassin box there is no imap server running to have an imap
to imap sync. Therefore I think imapsync can't handle this. offlineimap
seems to be interesting, because it seems to "dump" the mails to a local
folder, which could eaten by sa-learn. 

> there are tons more - Google "sync imap"

Before start scripting mysself I was looking for a smart and stable
script, which people use for long time without problems.

Ciao
Marcus

Re: sa-learn from a remote imap folder

Posted by Axb <ax...@gmail.com>.
On 09/12/2014 10:05 AM, Marcus Schopen wrote:
> Hi,
>
> spamassassin and imap (cyrus) are running on different boxes. What is
> best practice to learn spam from a remote imap folder? Any good working
> scripts?
>
> I found these:
>
> https://wiki.apache.org/spamassassin/RemoteImapFolder
>
> https://gist.github.com/colinmollenhour/4127743

Imapsync or OfflineIMAP

http://imapsync.lamiral.info/
http://offlineimap.org/

there are tons more - Google "sync imap"