SOUGHT 2.0 ?

[Axb <ax...@gmail.com>]

As most have probably noticed, the SOUGHT rules are not being publish/ 
updated anymore. (you can shutdown your updates)
The reasons for this are beyond this msg.

An option was to run such a project under the Apache umbrella but it 
makes it a VERY complicated process.

Thanks to Justin Mason which helped me a lot getting the bits and pieces 
glued together, I've been successfully autogenerating rules for $dayjob 
on a pretty regular basis.

It would be nice to be able to use this experience to replace the SOUGHT 
rules for everyone BUT:

- it's too much of project for me to run on my own.

- I don't really want to be the single point of failure.

- there's need of certain bits (DNS, CDN, etc) which are either above me 
or outside my interest.

- my spam data isn't enough to publish rules of "global relevance"

Question is: Can we (the SA users) get a "project" together with enough 
members taking care of different tasks to ensure that the project 
doesn't die when one person decides to step out?

What I "think" would be required:

- a project coordinator in charge of banging on tables and documenting 
the processes (I don't want this task).

- a closed mailing list for project members.

- lots more trap data / domains

- a DNS admin with a highly available DNS system to publish the update 
records.

- a packager which takes care of signing, and passing over updates to 
the CDN,etc.

- a CDN system for the sa-update clients to pickup the updates

I can offer 16 cores of fat iron to handle trap/spam/ham data and to 
run/babysit the rule generation process.

There's no way I can or want to handle all this on my own.

If you're interested, find you could cover one or more tasks and willing 
to provide long term commitment.

This is initial "brainstorming"...

Please post comments, etc on the SA list till we have a closed list.

And if you think I'm nuts.. I agree...

Axb

Re: SOUGHT 2.0 ?

[Ian Zimmerman <it...@buug.org>]

On Thu, 13 Nov 2014 09:28:30 -0000,
"Kevin Golding" <kp...@caomhin.org> wrote:

Kevin> The main thing that's going to be needed is good, reliable,
Kevin> data. We'll only get good rules with good feeds. That should be
Kevin> fairly low impact for people in many respects.

Kevin> Obviously there's always room to help with some code, so a bit of
Kevin> Perl or shell skills are a good thing. The impact of that on
Kevin> people will vary on how they work, but I doubt anyone will do
Kevin> anything to interfere with their running systems - as proven with
Kevin> masschecks it's fairly easy to sandbox things to one side for
Kevin> such analysis even if people do want to do anything on an
Kevin> important system.

Ok, I am still interested.  I'm a coder, my Perl is rusty but my shell
is current.  I can't provide trap servers but you'd be welcome to my
spam (all hand-verified by me).

-- 
Please *no* private copies of mailing list or newsgroup messages.
Local Variables:
mode:claws-external
End:

Re: SOUGHT 2.0 ?

[Noah Meyerhans <fr...@morgul.net>]

On Thu, Nov 13, 2014 at 02:08:30PM +0100, Axb wrote:
> >>As Alex has said there's a need for mirrors etc. - that could
> >>potentially be the biggest impact on volunteers (assuming they offer
> >>to help with that aspect) since they will be a more public facing
> >>contribution and it would be great if it didn't spend more time
> >>offline than online.
> >
> >What sort of disk space requirements? I'd be happy to run a mirror,
> >London or Hemel Hempsted, UK, so long as you don't need too many gigabytes.
> 
> 5 MB for rsync mirrors is more than enough.
> Rule files should not exceed 50KB or they hog SA performance
> Mirrors should be well connected (no volume restrictions). If rules
> become popular, they'll get thousands of requests/day.

I can help here, with ipv4 and v6 connected hosts at mit.edu and/or
linode.

> >Also could perhaps provide data if the process isn't too difficult to
> >set up. I don't have much mail throughput compared to some here, and
> >it's mostly UK-English-speaking, but I do have a variety of different
> >mail users.
> 
> We need trap domains which relay spam or point MX recs directly to a
> couple of specific servers - user reports are not reliable and can
> hadly provide enough data to make it worth the effort.

Can also help here, both with trap domains and mail processing, as
necessary.

I wonder if creating a separate mailing list for this project might make
sense at this point. Or do folks think we should keep working via the
main SA lists?

noah

Re: SOUGHT 2.0 ?

[Tom Johnson <tj...@mailroute.net>]

We (MailRoute) would be happy to pitch in.  We can host mirrors in a couple of datacenters (Los Angeles and Chicago). And we'll help out wherever else we can. 

Tom

Re: SOUGHT 2.0 ?

[Axb <ax...@gmail.com>]

On 11/13/2014 12:54 PM, Anthony Cartmell wrote:
>> As Alex has said there's a need for mirrors etc. - that could
>> potentially be the biggest impact on volunteers (assuming they offer
>> to help with that aspect) since they will be a more public facing
>> contribution and it would be great if it didn't spend more time
>> offline than online.
>
> What sort of disk space requirements? I'd be happy to run a mirror,
> London or Hemel Hempsted, UK, so long as you don't need too many gigabytes.

5 MB for rsync mirrors is more than enough.
Rule files should not exceed 50KB or they hog SA performance
Mirrors should be well connected (no volume restrictions). If rules 
become popular, they'll get thousands of requests/day.

> Also could perhaps provide data if the process isn't too difficult to
> set up. I don't have much mail throughput compared to some here, and
> it's mostly UK-English-speaking, but I do have a variety of different
> mail users.

We need trap domains which relay spam or point MX recs directly to a 
couple of specific servers - user reports are not reliable and can hadly 
provide enough data to make it worth the effort.

Re: SOUGHT 2.0 ?

[Anthony Cartmell <li...@fonant.com>]

> As Alex has said there's a need for mirrors etc. - that could  
> potentially be the biggest impact on volunteers (assuming they offer to  
> help with that aspect) since they will be a more public facing  
> contribution and it would be great if it didn't spend more time offline  
> than online.

What sort of disk space requirements? I'd be happy to run a mirror, London  
or Hemel Hempsted, UK, so long as you don't need too many gigabytes.

Also could perhaps provide data if the process isn't too difficult to set  
up. I don't have much mail throughput compared to some here, and it's  
mostly UK-English-speaking, but I do have a variety of different mail  
users.

Anthony
-- 
www.fonant.com - Quality web sites
Tel. 01903 867 810
Fonant Ltd is registered in England and Wales, company No. 7006596
Registered office: Amelia House, Crescent Road, Worthing, West Sussex,  
BN11 1QR

Re: SOUGHT 2.0 ?

[Kevin Golding <kp...@caomhin.org>]

On Thu, 13 Nov 2014 02:17:54 -0000, Ian Zimmerman <it...@buug.org> wrote:

> On Sat, 01 Nov 2014 10:06:57 -0000,
> "Kevin Golding" <kp...@caomhin.org> wrote:
>
> Kevin> So anyone else want to raise their hands?
>
> It depends.
>
> Would I mind a bit of regular maintenance work?  No, I wouldn't mind.
>
> Would I mind a major change in how I run my server - for instance,
> run a virus checker, or run the bleeding edge version of SA?  You
> betcha.  Not going to do that, sorry.
>
> So, I need more details before I raise my hand much above the keyboard  
> :-P

Well it's basically up to individuals how much and what they contribute,  
as things are still being planned it's hard to define some details on  
every task at the moment.

The main thing that's going to be needed is good, reliable, data. We'll  
only get good rules with good feeds. That should be fairly low impact for  
people in many respects.

Obviously there's always room to help with some code, so a bit of Perl or  
shell skills are a good thing. The impact of that on people will vary on  
how they work, but I doubt anyone will do anything to interfere with their  
running systems - as proven with masschecks it's fairly easy to sandbox  
things to one side for such analysis even if people do want to do anything  
on an important system.

As Alex has said there's a need for mirrors etc. - that could potentially  
be the biggest impact on volunteers (assuming they offer to help with that  
aspect) since they will be a more public facing contribution and it would  
be great if it didn't spend more time offline than online.

There's a big consideration of what people feel they might be able to  
offer really - if anyone knows someone with a spare Learjet sitting around  
they want to donate for the odd group meeting in their mansion there's a  
chance they'd get taken up on it. Actually that's a bit far-fetched, I  
think most of us like the spread of work and lack of need to find time to  
sit in the same room together to discuss every detail so if anyone is  
considering such an offer it would still need discussing, but the truth is  
it's not a "you must do this" type of volunteering it's a "well I can do  
that!" type.

Kevin

Re: SOUGHT 2.0 ?

[Ian Zimmerman <it...@buug.org>]

On Sat, 01 Nov 2014 10:06:57 -0000,
"Kevin Golding" <kp...@caomhin.org> wrote:

Kevin> So anyone else want to raise their hands?

It depends.

Would I mind a bit of regular maintenance work?  No, I wouldn't mind.

Would I mind a major change in how I run my server - for instance,
run a virus checker, or run the bleeding edge version of SA?  You
betcha.  Not going to do that, sorry.

So, I need more details before I raise my hand much above the keyboard :-P

Of course, I'd love to have the autogenerated rules back, so call me selfish.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Local Variables:
mode:claws-external
End:

Re: SOUGHT 2.0 ?

[Kevin Golding <kp...@caomhin.org>]

On Thu, 30 Oct 2014 20:10:22 -0000, Bob Proulx <bo...@proulx.com> wrote:

> Axb wrote:

>> It would be nice to be able to use this experience to replace the SOUGHT
>> rules for everyone BUT:
>> ...
>
> All very good, reasonable and understandable reasons.  And those
> reasons also aply to me too.  Insert all of those reasons into why I
> didn't personally jump forward.

Life has many annoying ways of getting in the way of some great ideas, and
it's why an individual leaping forward has complications, however...

>> Question is: Can we (the SA users) get a "project" together with enough
>> members taking care of different tasks to ensure that the project  
>> doesn't
>> die when one person decides to step out?

Yes we can!

> I was disappointed to see that there wasn't a bunch of motivated
> people jumping in with both feet to maintain this.  I totally
> understand everyone being in the same situation of never having enough
> time and resources to volunteer for something like this type of
> project.  I am right there with you.
>
> I didn't want to see the option whither and die without voicing a
> positive cheer for it.  Perhaps some more discussion would motivate
> more people to become involved.

Well I think there are some legs to this too.

I've been talking to Alex about some of the aspects and things I can do do
to help. Like most of us I don't want to sign up for something that would  
be a monster resting purely on my shoulders, but having gone over a few  
bits and pieces it seems very manageable with just a bit of organisation  
and coordination. Which is why I've offered to throw my hat in the ring  
and do various bits to make this work.

I really think that's the key aspect too, obviously we all share an  
interest and there's an overlap of skillsets too, which means once you  
start taking apart what's planned it becomes a lot less commitment for  
each of us. Alex auto generates rules for himself already, I have  
something I suspect is less sophisticated in place for my needs, we're  
seeing interest from other people already doing this kind of thing and  
those that aren't are pretty close in the various things we know they are  
doing. All in all it looks like it is close to being very viable.

The big thing that should make this worth pursuing is the somewhat  
distributed nature of it all. I know in what we've discussed so far it's  
looking like we'll be able to avoid the single point of failure issue, and  
more importantly in terms of volunteering, the single point of pressure.  
Once set-up it's largely just a case of maintenance, and with people being  
involved we should be able to jump on a world cruise and be happy that we  
won't be woken somewhere near the Equator with a problem - there's  
redundancy forming already so I'll really emphasise that it won't be a  
huge burden on anyone. None of us already discussing this want that for  
ourselves and we don't want to push it on others either - we've got some  
experience of spreading the load across teams to make lives easier and  
it's basically a case of a little extra care and attention in planning  
should make the whole thing run smoothly.

So anyone else want to raise their hands?

Re: SOUGHT 2.0 ?

[Bob Proulx <bo...@proulx.com>]

Axb wrote:
> As most have probably noticed, the SOUGHT rules are not being publish/
> updated anymore. (you can shutdown your updates)

Yes.  A shame since they were quite effective here at tagging spam.

> An option was to run such a project under the Apache umbrella but it makes
> it a VERY complicated process.
> 
> Thanks to Justin Mason which helped me a lot getting the bits and pieces
> glued together, I've been successfully autogenerating rules for $dayjob on a
> pretty regular basis.

Cool!

> It would be nice to be able to use this experience to replace the SOUGHT
> rules for everyone BUT:
> ...

All very good, reasonable and understandable reasons.  And those
reasons also aply to me too.  Insert all of those reasons into why I
didn't personally jump forward.

> Question is: Can we (the SA users) get a "project" together with enough
> members taking care of different tasks to ensure that the project doesn't
> die when one person decides to step out?

I was disappointed to see that there wasn't a bunch of motivated
people jumping in with both feet to maintain this.  I totally
understand everyone being in the same situation of never having enough
time and resources to volunteer for something like this type of
project.  I am right there with you.

I didn't want to see the option whither and die without voicing a
positive cheer for it.  Perhaps some more discussion would motivate
more people to become involved.

> This is initial "brainstorming"...

Thank you very much for raising this topic for discussion.  If these
were available I would use them.  If there is some way that I could
help I would be very happy to do so.

Bob