You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Shivram Krishnan <ro...@gmail.com> on 2016/06/29 00:33:00 UTC
Corpus of Spam/Ham headers(Source IP) for research
Hey Guys,
I am a researcher at the University of Southern California (
https://steel.isi.edu/ ), and I have been working on making Blacklists
more effective by combining different sources of Blacklists, and creating a
Blacklists specific for a particular network.
Though I have devised a mechanism to generate these blacklists, I am not
finding a suitable evaluation metric. It would be great if somebody could
give me a dataset of source IP addresses of emails received by your network
which have been marked as HAM/SPAM by Spamassassin for the year 2016. I do
not require the entire SPAM/HAM emails. Using this , I could evaluate the
false positives/ true positives of my technique.
I had posted a similar question in the previous thread, but that did not
get much response.
Looking forward to your replies!
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Rob McEwen <ro...@invaluement.com>.
On 6/29/2016 1:00 AM, Shivram Krishnan wrote:
> Thank you so much for your views. I agree that your customers would not
> like it if you share information. But Oliver suggested , I need only the
> source IP addresses of the Spam and Ham emails , which can even be
> anonymized in the last octet.
Unfortunately, accuracy and credibility goes down since there then isn't
any easy way to audit or double-check the root cause of the classification.
For example, some people classify spam as "what our filter said was
spam" and ham as "what our filter said was ham". For most well-run
systems, that is going to be overall very accurate. But there can still
be egregious mistakes. And assuming that the existing filter is 100%
accurate leaves no room for improvement. It also has the unfortunate
side effect of rubber stamping the most elusive spams, sent by the
shrewdest of spammers, as ham.
If an anti-spam blacklist comes along that is very good at blocking
messages that are unsolicited and not desired by end users... but sent
by the most shrewd spammer who evade lists like SpamHaus and SURBL (at
least for some time)... and where the collateral damage for listing such
domains and sending IPs is non-existent... such a blacklist might STILL
fare badly in such a rating system... which would then MISTAKENLY assume
that such a blacklist has many False Positives.
Stats collected from user complaints about False Negatives can also be
helpful. However, for snowshoe spam, that is often a lagging
indicator... sometimes days behind reality--where the spammer has
already moved to new domains/IPs--but such could help such a ratings
system to make wise adjustments to past ham/spam stats.
Hijacked IP and domains is another sticky issue. Over the past several
years, this has become epidemic! If the volume of legit usage is
relatively low, and the IP or domain has been hijacked by a spammer...
then at SOME point, an anti-spam blacklist should not be penalized for
listing such. In fact, Spamhaus does this frequently (lists hijacked
domains/IPs where the cost/benefit ratio for that listing is well
justified). Some other lists also blacklist hijacked domains/IPs... but
are often not as good at making proper cost/benefit ratio decisions...
where they list somewhat large senders who had a somewhat small and
short-lived spam outbreak. Finding a way to penalize or reward the lists
that block hijacked domains/IPs that Spamhaus misses, based on whether
they do (or don't do) a good job of making overall good decisions about
the cost/benefit ration of a potential listing's collateral damage... is
also tricky.
My main point is... how to reward blacklists that are more accurate, but
without penalizing them for not being a redundant copy of Zen. It isn't
as easy as it sounds in a ratings system. (even if real life usage of
such by a hoster or ISP can quickly lead to fewer complains from
customers about about FP and FNs)
--
Rob McEwen
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Antony Stone <An...@spamassassin.open.source.it>.
On Wednesday 29 June 2016 at 17:38:35, Shivram Krishnan wrote:
> There has been enough research which has been done in this field were the
> authors have obtained the data from network operators. This
> <http://repository.upenn.edu/cgi/viewcontent.cgi?article=1962&context=cis_r
> eports> for instance is a paper from UPenn, which has collected over 31
> million Mail Headers (not only IP address) to validate their method.
Page 6:
"For testing purposes, we procured approximately 31 million email headers
collected at the University of Pennsylvania engineering email servers..."
So, I repeat the suggestion from my previous reply - go and ask your own
university's mail admins.
Antony.
--
Behind the counter a boy with a shaven head stared vacantly into space,
a dozen spikes of microsoft protruding from the socket behind his ear.
- William Gibson, Neuromancer (1984)
Please reply to the list;
please *don't* CC me.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Joe Quinn <jq...@pccc.com>.
On 6/29/2016 11:50 AM, Shivram Krishnan wrote:
> Hello Antony,
>
> We will be getting headers from our University. The only reason why we
> want other list is that we are tailoring Blacklists for specific
> networks, to see how these blacklists perform. The idea being , your
> network may not be seeing the same attack vectors as what the USC
> network sees.
>
>
> Also getting the IP's in anonymized last octet would also help , as we
> are creating Blacklists in terms of Prefixes.
>
You should look at what masscheck does. Instead of uploading messages to
get tested, masscheckers run the rules against their corpus locally and
upload the match set.
Provide a mechanism for people to generate their own results, which they
can upload with absolutely no identifying information.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by David Jones <dj...@ena.com>.
>From: Shivram Krishnan <ro...@gmail.com>
>Sent: Wednesday, June 29, 2016 10:50 AM
>To: Antony Stone
>Cc: users@spamassassin.apache.org
>Subject: Re: Corpus of Spam/Ham headers(Source IP) for research
>Hello Antony,
>We will be getting headers from our University. The only reason why we want other list is that we are tailoring Blacklists for specific networks, to see how these blacklists perform. The >idea being , your network may not be seeing the same attack vectors as what the USC network sees.
http://ren-isac.net/
Would this be a better source for your information? I don't think anyone on this list
has spare time to extract this information for you even if we had no other reservations
about sharing the data.
>Also getting the IP's in anonymized last octet would also help , as we are creating Blacklists in terms of Prefixes.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Shivram Krishnan <ro...@gmail.com>.
Hello Antony,
We will be getting headers from our University. The only reason why we want
other list is that we are tailoring Blacklists for specific networks, to
see how these blacklists perform. The idea being , your network may not be
seeing the same attack vectors as what the USC network sees.
Also getting the IP's in anonymized last octet would also help , as we are
creating Blacklists in terms of Prefixes.
On Wed, Jun 29, 2016 at 8:41 AM, Antony Stone <
Antony.Stone@spamassassin.open.source.it> wrote:
> On Wednesday 29 June 2016 at 17:35:28, Shivram Krishnan wrote:
>
> > We could solve this problem , If you could submit the set of IP's by
> > anonymising the last octet of the IP addresses.
>
> What good is that going to do you in your research project?
>
> > Also we could sign an NDA (if you are willing to provide the list) , so
> > that I would not make the list of IP's public.
>
> We are not concerned about you making it *public* - we are concerned about
> the
> list somehow getting into the hands of untrustworthy individuals who would
> make undesirable use of it.
>
> If that were to happen, there is no way we could trace that use back to you
> and make a claim under the NDA, therefore it is no use.
>
> Have you considered askng your university mail admins for such a list, from
> *their* servers? I'm sure they process quite a good amount of email every
> year.
>
>
> Antony.
>
> --
> The words "e pluribus unum" on the Great Seal of the United States are
> from a
> poem by Virgil entitled "Moretum", which is about cheese and garlic salad
> dressing.
>
> Please reply to the
> list;
> please *don't* CC
> me.
>
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Antony Stone <An...@spamassassin.open.source.it>.
On Wednesday 29 June 2016 at 17:35:28, Shivram Krishnan wrote:
> We could solve this problem , If you could submit the set of IP's by
> anonymising the last octet of the IP addresses.
What good is that going to do you in your research project?
> Also we could sign an NDA (if you are willing to provide the list) , so
> that I would not make the list of IP's public.
We are not concerned about you making it *public* - we are concerned about the
list somehow getting into the hands of untrustworthy individuals who would
make undesirable use of it.
If that were to happen, there is no way we could trace that use back to you
and make a claim under the NDA, therefore it is no use.
Have you considered askng your university mail admins for such a list, from
*their* servers? I'm sure they process quite a good amount of email every
year.
Antony.
--
The words "e pluribus unum" on the Great Seal of the United States are from a
poem by Virgil entitled "Moretum", which is about cheese and garlic salad
dressing.
Please reply to the list;
please *don't* CC me.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 29 Jun 2016, at 11:38, Shivram Krishnan wrote:
> Hello Bill,
>
> There has been enough research which has been done in this field were
> the
> authors have obtained the data from network operators. This
> <http://repository.upenn.edu/cgi/viewcontent.cgi?article=1962&context=cis_reports>
> for
> instance is a paper from UPenn, which has collected over 31 million
> Mail
> Headers (not only IP address) to validate their method.
As Anthony has pointed out, they got those from their own mail system.
One mail system, coherent mail filtering practices, administrators with
powerful disincentives to falsify data, and enough detail in the data
that it would be extremely difficult to significantly falsify without
being obvious.
> We are trying to get HAM/SPAM lists from different networks, to
> validate
> our technique, which curates Blacklists for specific Network.
I understand that. Unfortunately, you are trying to get data in a way
that CANNOT get you trustworthy or even coherent data. Mail filtering
statistics between different systems wouldn't just differ
quantitatively, they would would be qualitatively different in scope and
meaning. I help run multiple mail systems, each of which has a unique
profile of how much and what sorts of mail reaches the point of
SpamAssassin scoring (always the final line of defense) and the
mechanisms by which attempts to send mail are stopped before the SMTP
DATA command. In some cases, substantial shunning of address space known
to be controlled by spammers happens in border routers and isn't really
countable; even if we had persistent logs of every intentional port 25
SYN packet drop, I couldn't know whether a trio of those from one IP in
a minute represents one spam attempt or three.
Even stipulating that no one would feed you intentionally false data (a
very dubious stipulation) if you received data from a diverse set of
mail systems you would be getting datasets with divergent semantics.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Shivram Krishnan <ro...@gmail.com>.
Hello Bill,
There has been enough research which has been done in this field were the
authors have obtained the data from network operators. This
<http://repository.upenn.edu/cgi/viewcontent.cgi?article=1962&context=cis_reports>
for
instance is a paper from UPenn, which has collected over 31 million Mail
Headers (not only IP address) to validate their method.
We are trying to get HAM/SPAM lists from different networks, to validate
our technique, which curates Blacklists for specific Network.
On Wed, Jun 29, 2016 at 8:02 AM, Bill Cole <
sausers-20150205@billmail.scconsult.com> wrote:
> On 29 Jun 2016, at 1:00, Shivram Krishnan wrote:
>
> Hello Bill,
>>
>> Thank you so much for your views. I agree that your customers would not
>> like it if you share information. But Oliver suggested , I need only the
>> source IP addresses of the Spam and Ham emails , which can even be
>> anonymized in the last octet.
>>
>> Will that still be a privacy concern?
>>
>
> No, but there would still be a data collection and preparation cost that
> is substantial and a fundamental study design problem: you have no controls
> for data validity or sampling issues.
>
> In total honesty: if your approach to this research has been cleared by
> your faculty advisor and not stopped, that advisor is either incompetent or
> is intentionally sabotaging you. You cannot gather a valid data set this
> way and the data you are asking for cannot even be verified to be anything
> other than pure invention. If your advisor does not see that, they are in
> the wrong profession.
>
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Shivram Krishnan <ro...@gmail.com>.
Hey guys,
I see there is a lot of concern of revealing the set of Spam IP's and Ham
IP's , where one could get to know either a customer - company relation
(which may be private) and might generate suphosticated phishing attacks.
We could solve this problem , If you could submit the set of IP's by
anonymising the last octet of the IP addresses.
Also we could sign an NDA (if you are willing to provide the list) , so
that I would not make the list of IP's public.
On Wed, Jun 29, 2016 at 8:02 AM, Bill Cole <
sausers-20150205@billmail.scconsult.com> wrote:
> On 29 Jun 2016, at 1:00, Shivram Krishnan wrote:
>
> Hello Bill,
>>
>> Thank you so much for your views. I agree that your customers would not
>> like it if you share information. But Oliver suggested , I need only the
>> source IP addresses of the Spam and Ham emails , which can even be
>> anonymized in the last octet.
>>
>> Will that still be a privacy concern?
>>
>
> No, but there would still be a data collection and preparation cost that
> is substantial and a fundamental study design problem: you have no controls
> for data validity or sampling issues.
>
> In total honesty: if your approach to this research has been cleared by
> your faculty advisor and not stopped, that advisor is either incompetent or
> is intentionally sabotaging you. You cannot gather a valid data set this
> way and the data you are asking for cannot even be verified to be anything
> other than pure invention. If your advisor does not see that, they are in
> the wrong profession.
>
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 29 Jun 2016, at 1:00, Shivram Krishnan wrote:
> Hello Bill,
>
> Thank you so much for your views. I agree that your customers would
> not
> like it if you share information. But Oliver suggested , I need only
> the
> source IP addresses of the Spam and Ham emails , which can even be
> anonymized in the last octet.
>
> Will that still be a privacy concern?
No, but there would still be a data collection and preparation cost that
is substantial and a fundamental study design problem: you have no
controls for data validity or sampling issues.
In total honesty: if your approach to this research has been cleared by
your faculty advisor and not stopped, that advisor is either incompetent
or is intentionally sabotaging you. You cannot gather a valid data set
this way and the data you are asking for cannot even be verified to be
anything other than pure invention. If your advisor does not see that,
they are in the wrong profession.
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Shivram Krishnan <ro...@gmail.com>.
Hello Bill,
Thank you so much for your views. I agree that your customers would not
like it if you share information. But Oliver suggested , I need only the
source IP addresses of the Spam and Ham emails , which can even be
anonymized in the last octet.
Will that still be a privacy concern?
On Tue, Jun 28, 2016 at 9:04 PM, Bill Cole <
sausers-20150205@billmail.scconsult.com> wrote:
> On 28 Jun 2016, at 20:33, Shivram Krishnan wrote:
>
> Hey Guys,
>>
>> I am a researcher at the University of Southern California (
>> https://steel.isi.edu/ ), and I have been working on making Blacklists
>> more effective by combining different sources of Blacklists, and creating
>> a
>> Blacklists specific for a particular network.
>>
>> Though I have devised a mechanism to generate these blacklists, I am not
>> finding a suitable evaluation metric. It would be great if somebody could
>> give me a dataset of source IP addresses of emails received by your
>> network
>> which have been marked as HAM/SPAM by Spamassassin for the year 2016. I
>> do
>> not require the entire SPAM/HAM emails. Using this , I could evaluate the
>> false positives/ true positives of my technique.
>>
>> I had posted a similar question in the previous thread, but that did not
>> get much response.
>>
>> Looking forward to your replies!
>>
>
> You may be disappointed...
>
> It is extremely difficult to get a diverse corpus of non-bulk "Ham" for
> any form of research because most people consider such mail private, even
> when it is not very well protected in practice. I help run mail systems for
> a number of small and medium sized businesses, which handle thousands of
> pieces of one-to-one mail and about as much non-spam B2B bulk mail every
> day. It is functionally impossible for me to try to provide a researcher
> any of that due to real personal privacy and business confidentiality
> issues and the hard cost of the simple logistics of getting permission and
> gathering the mail. Consumer ISPs would be even less able and willing than
> corporate mail operators, because most of them have highly impersonal
> relationships with their customers and have staff to users ratios that make
> such a project inconceivable without necessarily deceiving users: their
> only way would be to hide permission deep in a TOS no one reads.
>
> The simple bottom line: Mail system operators are not able to give you the
> corpus you need. It's not that we wouldn't like to be able to, but we don't
> have it assembled because we can't ethically do so without investing huge
> amounts of time we do not have.
>
> HOWEVER...
>
> USC is in a particularly good position to be collecting such a corpus. "
> isi.edu" and "usc.edu are two of the oldest domains on the Internet and
> unless they are very strange among such domains, they are targeted by huge
> streams of spam. Less noticeable: they are probably targeted by substantial
> streams of non-spam "oops" mail. I'd bet that Jon Postel's old address gets
> targeted by hundreds of innocent messages every day, along with thousands
> of pieces of spam. Someone could hand-classify that. You also have the
> advantage of being a sizable private university, so you aren't limited by
> some of the rules public universities have, and you could probably do more
> than a public school -- for example, UCLA or UCB -- could to encourage
> students and staff to let you do research on their Ham.
>
> IOW: MY users would resist or refuse me handing you their private and/or
> confidential mail for "research" they don't care about, I can't quickly
> explain, and none of us is being paid to help. OTOH: YOUR institution is
> well-situated to generate exactly the corpus you seek.
>
Re: Corpus of Spam/Ham headers(Source IP) for research
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 28 Jun 2016, at 20:33, Shivram Krishnan wrote:
> Hey Guys,
>
> I am a researcher at the University of Southern California (
> https://steel.isi.edu/ ), and I have been working on making
> Blacklists
> more effective by combining different sources of Blacklists, and
> creating a
> Blacklists specific for a particular network.
>
> Though I have devised a mechanism to generate these blacklists, I am
> not
> finding a suitable evaluation metric. It would be great if somebody
> could
> give me a dataset of source IP addresses of emails received by your
> network
> which have been marked as HAM/SPAM by Spamassassin for the year 2016.
> I do
> not require the entire SPAM/HAM emails. Using this , I could evaluate
> the
> false positives/ true positives of my technique.
>
> I had posted a similar question in the previous thread, but that did
> not
> get much response.
>
> Looking forward to your replies!
You may be disappointed...
It is extremely difficult to get a diverse corpus of non-bulk "Ham" for
any form of research because most people consider such mail private,
even when it is not very well protected in practice. I help run mail
systems for a number of small and medium sized businesses, which handle
thousands of pieces of one-to-one mail and about as much non-spam B2B
bulk mail every day. It is functionally impossible for me to try to
provide a researcher any of that due to real personal privacy and
business confidentiality issues and the hard cost of the simple
logistics of getting permission and gathering the mail. Consumer ISPs
would be even less able and willing than corporate mail operators,
because most of them have highly impersonal relationships with their
customers and have staff to users ratios that make such a project
inconceivable without necessarily deceiving users: their only way would
be to hide permission deep in a TOS no one reads.
The simple bottom line: Mail system operators are not able to give you
the corpus you need. It's not that we wouldn't like to be able to, but
we don't have it assembled because we can't ethically do so without
investing huge amounts of time we do not have.
HOWEVER...
USC is in a particularly good position to be collecting such a corpus.
"isi.edu" and "usc.edu are two of the oldest domains on the Internet and
unless they are very strange among such domains, they are targeted by
huge streams of spam. Less noticeable: they are probably targeted by
substantial streams of non-spam "oops" mail. I'd bet that Jon Postel's
old address gets targeted by hundreds of innocent messages every day,
along with thousands of pieces of spam. Someone could hand-classify
that. You also have the advantage of being a sizable private university,
so you aren't limited by some of the rules public universities have, and
you could probably do more than a public school -- for example, UCLA or
UCB -- could to encourage students and staff to let you do research on
their Ham.
IOW: MY users would resist or refuse me handing you their private and/or
confidential mail for "research" they don't care about, I can't quickly
explain, and none of us is being paid to help. OTOH: YOUR institution is
well-situated to generate exactly the corpus you seek.