The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

RE: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

WARNING FOR ESPs AND MARKETERS: Google's "goo.gl" shortner is OUT OF 
CONTROL.

Spammers are starting to use this to evade spam filters, and Google 
isn't keeping up with the abuse, nor shutting these down fast enough. 
Along with blackhat spammers, we're seeing evidence that many 
gray-ish-hat spammers are jumping on this bandwagon, in the hopes that 
they will get more mail delivered if they can avoid using their domain 
in the clickable links. Keep in mind that, if a marketer is doing things 
the right way, they should have no need to obfuscate their own domain 
name. They should instead proudly use it and not feel the need to hide 
behind Google's shortner. Yes, there are many legitimate uses of 
Google's shortner, too. However, we are now at a point where a VERY 
large % (a majority?) of uses of these headed to a typical user's 
mailbox are egregious spams, and a significant additional portion are 
likely-spams. THEREFORE: If you like having NOT-blacklisted IPs, be 
advised that the invaluement anti-spam DNSBL system is now adding "bad" 
points to the scoring of all messages that use the "goo.gl" shortner, 
and we're amplifying other "bad" points. (We're also doing various 
sophisticated things to minimize potential resulting FPs, too. But this 
will still put MANY marginal IPs and domains into our blacklists that 
normally might have barely avoided an invaluement blacklisting!)

YOU HAVE BEEN WARNED! DON'T ALLOW YOUR CLIENTS TO DO THIS!

ALSO: We're very seriously evaluating the option of converting each 
shortner to the URL it redirects to - and then potentially starting to 
add those domains or IPs within those URLs to our ivmURI domain/URI 
blacklist. This might not cause other such messages to get blocked, but 
it will have other negative repercussions for other uses of that domain.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 3/15/2018 11:13 AM, shanew@shanew.net wrote:
> You might take a look at
> https://developers.google.com/url-shortener/v1/getting_started
>
> 1 millllion requests per day is the default limit.


Excellent! Thanks for the suggestion. This should help me MUCH!

But, unfortunately, it still leaves a lot to be desired for fixing the 
problems I described... when it comes to more distribution of automated 
"plugins" or having spam filters automatically doing this - in a "set it 
and forget it mode".

Still, there is a good argument that if someone has a high enough volume 
to trigger rate limiting... then they ought to have a large enough staff 
to handle going the extra mile and setting up API access systems, etc. 
But I wish it could be more simple. MANY are going to struggle with 
this... and NOT easily know all the details discussed on this thread!

But again, this should help me (and others) much... and it is good to 
know that there is a proper way to do this at a higher volume that meets 
Google's approval.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[sh...@shanew.net]

You might take a look at
https://developers.google.com/url-shortener/v1/getting_started

1 millllion requests per day is the default limit.

On Wed, 14 Mar 2018, Rob McEwen wrote:

> On 2/20/2018 9:42 PM, Rob McEwen wrote:
>       Google might easily start putting captchas in the way or
>       otherwise consider such lookups to be abusive and/or mistake
>       them for malicious bots...
> 
> This prediction turned out to be 100% true. Even though others have
> mentioned that they have been able to do high-volume lookups with no
> problems... And granted I wasn't implementing a multi-server or multi-ip
> lookup strategy... But I don't think I was doing nearly as many lookups as
> others have claimed that they were able to do. I took a batch of 55,000
> spams that I had collected from the past 4 weeks where those spams were
> maliciously using the Google shortener as a way to get their spam delivered
> via hiding their spammy domain names from spam filters. I started checking
> those by looking up the redirect from Google's redirector, but without
> actually visiting the site that the redirector was pointing to. Please note
> that I was doing the lookups one-at-a-time, not starting the next lookup
> until the last lookup had completed. After about ONLY 1,400 lookups, ALL of
> my following lookups started hitting captchas. See attached screenshot.
> Also, other than not sending from multiple IPs, I was otherwise doing
> everything correct to make my script look/act like a regular browser.
> 
> I'll try spreading it out between multiple IPs in order to try to avoid rate
> limits... However... This is still cause for concern about high-volume
> lookups in high production systems... those may have to be implemented a
> little more carefully if they're going to do these kind of lookups!
> 
> Just because small or medium production systems are able to do this... Or
> just because somebody went out of their way to get more sophisticated with
> it to get it to work out... doesn't mean that it's going to work in high
> production systems that are trying to use "canned" software or plugins. This
> is a particular challenge for anti-spam blacklists because they typically
> process a very high volume of spams. Hopefully, the randomness of the ones I
> process as they come in... will be sufficiently spread out enough to avoid
> rate limiting?
> 
> It was my hope to start processing these live with my own DNSBL engine, so
> that I could start blacklisting the domains that they redirect to... In
> those cases where they were not already blacklisted... Now I'm going to have
> to deal with constantly trying to make sure that I'm not hitting this
> captcha, along with implementing some other strategies to hopefully prevent
> that.
> 
> But this brings up a whole other issue... That is more of a policy or legal
> issue... is Google basically making a statement that automated lookups are
> not welcome? Or are considered abusive?
> 
> (btw, I could have collected order of magnitudes more than 55,000 of THESE
> types of spams, but this was merely what was left over in an after-the-fact
> search of my archives, after a lot of otherwise redundant spams had already
> been purged from my system.)
> 
> PS - Once I gather this information, I will submit more details about the
> results of this testing. But what is shocking right now is that less than
> four tenths of 1% of these redirect URLs has been terminated, even though
> they average two weeks old, with some almost a month old.
> 
>

-- 
Public key #7BBC68D9 at            |                 Shane Williams
http://pgp.mit.edu/                |      System Admin - UT CompSci
=----------------------------------+-------------------------------
All syllogisms contain three lines |              shanew@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/20/2018 9:42 PM, Rob McEwen wrote:
> Google might easily start putting captchas in the way or otherwise 
> consider such lookups to be abusive and/or mistake them for malicious 
> bots...

This prediction turned out to be 100% true. Even though others have 
mentioned that they have been able to do high-volume lookups with no 
problems... And granted I wasn't implementing a multi-server or multi-ip 
lookup strategy... But I don't think I was doing nearly as many lookups 
as others have claimed that they were able to do. I took a batch of 
55,000 spams that I had collected from the past 4 weeks where those 
spams were maliciously using the Google shortener as a way to get their 
spam delivered via hiding their spammy domain names from spam filters. I 
started checking those by looking up the redirect from Google's 
redirector, but without actually visiting the site that the redirector 
was pointing to. Please note that I was doing the lookups one-at-a-time, 
not starting the next lookup until the last lookup had completed. After 
about ONLY 1,400 lookups, ALL of my following lookups started hitting 
captchas. See attached screenshot. Also, other than not sending from 
multiple IPs, I was otherwise doing everything correct to make my script 
look/act like a regular browser.

I'll try spreading it out between multiple IPs in order to try to avoid 
rate limits... However... This is still cause for concern about 
high-volume lookups in high production systems... those may have to be 
implemented a little more carefully if they're going to do these kind of 
lookups!

Just because small or medium production systems are able to do this... 
Or just because somebody went out of their way to get more sophisticated 
with it to get it to work out... doesn't mean that it's going to work in 
high production systems that are trying to use "canned" software or 
plugins. This is a particular challenge for anti-spam blacklists because 
they typically process a very high volume of spams. Hopefully, the 
randomness of the ones I process as they come in... will be sufficiently 
spread out enough to avoid rate limiting?

It was my hope to start processing these live with my own DNSBL engine, 
so that I could start blacklisting the domains that they redirect to... 
In those cases where they were not already blacklisted... Now I'm going 
to have to deal with constantly trying to make sure that I'm not hitting 
this captcha, along with implementing some other strategies to hopefully 
prevent that.

But this brings up a whole other issue... That is more of a policy or 
legal issue... is Google basically making a statement that automated 
lookups are not welcome? Or are considered abusive?

(btw, I could have collected order of magnitudes more than 55,000 of 
THESE types of spams, but this was merely what was left over in an 
after-the-fact search of my archives, after a lot of otherwise redundant 
spams had already been purged from my system.)

PS - Once I gather this information, I will submit more details about 
the results of this testing. But what is shocking right now is that less 
than four tenths of 1% of these redirect URLs has been terminated, even 
though they average two weeks old, with some almost a month old.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 1:17 AM, @lbutlr wrote:
> goo.gl (and other shorteners) are used for far more than email.
> That said, most my incoming email is rejected long before it get to 
> any sort of URI lookups based on just the transaction information, 
> That is to say, upwards of 90% of incoming mail is rejected before DATA. 

That is besides the point. I'm discussing the more elusive spams that 
often slip past spam filters and are difficult to block. I'm not very 
concerned with the "low hanging fruit" that you're describing - that is 
ANOTHER topic!

>> 286 total spams blocked that had a shortner,
> That's not enough to have any sort of reliable statistical data.

This is what was pulled from a corpus of 5K spams. Are you sure you 
didn't get confused and think I said that my total corpus I searched 
only involved 286 spams to start with?

Also, yes, different data batches from different sources can have 
different idiosyncrasies - but to discount this as not reliable due to 
being too small of sample size - is laughable. The server has a good 
variety of dozens of different domains and thousand of different users. 
These are actual "brick and mortar" businesses in the US with real 
people: Schools, law firms, real estate companies, manufacturing 
companies, service-oriented companies - so there is a SUBSTANTIAL 
variety. At the same time, I've had a distinct uptick in complaints 
about spam where the goo.gl shortner was a large factor in recent 
unblocked spams complaints - where those complaints are normally few and 
far between. With changes I've made in the past few days, the number of 
such spams slipping past my filter was been sharply reduced. Also, over 
the years, I've noticed that patterns I seen on my server, like this 
one, often match patterns that others report. It defies Occam's Razor to 
suggestion that I have this magical concentration of egregious goo.gl 
shortner spams that is magically hitting these very diverse and 
unrelated companies for whom I host mail, and not hitting elsewhere. 
That is how statistics tend to work. The next time you read about a poll 
that a newpaper or news show conducted, you may be shocked to learn that 
this poll that gave a nationwide estimate with a small margin of error - 
often only had about a thousand participants. My study isn't as 
scientific because my users aren't perfectly random. But it is diverse 
enough to simulate a sufficient amount of randomness so as to be an 
extremely good indicator. Likewise, whenever I hear about a new spam 
trend that is effecting many people, I pretty much never come up empty 
when searching for examples of that hitting my servers.

>> 187 total legit messages had a hit on at least one of hundreds of URL shortners
> So the use of a shortner is a poor spam indicator. Even in your corpus, and a negligible indicator even when specifically looking at goo.gl

You have an amazing talent for missing the point. Most often, when used 
in legit mail... it was more of an afterthought - such as someone 
sending a link to a Google maps reference - but yet where the sender was 
NOT using the shortner for obfuscation of spammer's domains, or even for 
their mail hyperlinks. But THAT is what happened when these were used in 
spams, where it was used to HIDE identity by obfuscating their main 
links. So there is a strong correlation right there. But you want an 
"all or none" strawman to argue against, while you conveniently ignore 
many points I've made. Yes - it is frustrating that it is NOT extremely 
easy to differentiate the ones who use it for obfuscation from the ones 
have more appropriate usages. But there are ways. This is why I had to 
spend some hours making adjustments to my system - not some minutes. You 
make a good point that it is unwise to outright block ALL messages due 
to them having a shortner, or having the goo.gl shortner - but I'm light 
years more sophisticated than that - and I haven't argued that even one 
time in this thread.

Another problem with your statement is that if this loophole isn't 
addressed it will only get larger. Also, many certain of these spams are 
already very difficult to block - but would be very easy to block if the 
spammer wasn't hiding behind the shortner. That is a real problem that 
STILL exists regardless of how often shortners are used in legit mail - 
and regardless of how often they are used in spams - but you don't want 
to face that truth, it seems?

The BOTTOM LINE is that anything that (a) helps blacklist those senders 
who are doing this obfuscation in order to facilitate sending spam -AND- 
(b) motivate marketers and ESPs (& their clients) to avoid this tactic - 
is a step in the right direction.

>> Google's shortner is DOMINATING in its spam usage, where 92% (262 of 286) of ALL spam that contained shortners used Google.
> But about 25% of goo.gl containing email is not spam, by your own numbers. So, a very poor metric.


A poor metric if only idiots ran spam filters and DNSBLs. But, 
thankfully, I'm able to separate the legit uses - from those spammers 
who are trying to hide their identities in order to facilitate spamming 
- a point lost on you. I've NEVER ONCE argued for creating rules that 
blocked ALL messages that used shortners - but that is the "straw man" 
that you're arguing against. This is also one of the very reasons that 
spammers are so happy with this loophole - they know that nobody is 
going to create a rule that blocks goo.gl (or even scores that high 
against it). But thankfully, they'll get SOME relief as more of those 
who abuse it find their IPs blacklisted at invaluement (and hopefully 
other places), and rightly so! (yet without the collateral damage you 
keep predicting or implying)

If we're going to continue this conversation, can you please STOP 
putting words in my mouth and arguing against "straw men"? Also, I 
understand your very valid concerns about collateral damage. I've 
addressed that numerous times and in numerous ways, in numerous posts. 
This is getting tiresome.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-20 (19:42 MST), Rob McEwen <ro...@invaluement.com> wrote:
> 
> I ran stats on a sample set of a few thousand mailboxes, over a period of several hours today (mostly during business hours for these particular organizations who use these mailboxes) - and this produced a combined 24K legit messages, and 5K spams (I'm guessing that most systems have more spams per amount of hams? But those were the numbers for this server.)

goo.gl (and other shorteners) are used for far more than email.

That said, most my incoming email is rejected long before it get to any sort of URI lookups based on just the transaction information, That is to say, upwards of 90% of incoming mail is rejected before DATA.

> 286 total spams blocked that had a shortner,

That's not enough to have any sort of reliable statistical data.

> 187 total legit messages had a hit on at least one of hundreds of URL shortners 

So the use of a shortner is a poor spam indicator. Even in your corpus, and a negligible indicator even when specifically looking at goo.gl.

> Google's shortner is DOMINATING in its spam usage, where 92% (262 of 286) of ALL spam that contained shortners used Google. 

But about 25% of goo.gl containing email is not spam, by your own numbers. So, a very poor metric.

-- 
"You can speak soon and write like a graduate college if me let you help
for a day of 15 minutes" "1963" Issue #1

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/20/2018 6:05 PM, @lbutlr wrote:
> On 2018-02-20 (08:30 MST), Rob McEwen <ro...@invaluement.com> wrote:
>> Spammers are starting to use this to evade spam filters,
> This is not news. Spammers have been using shortness since 3 seconds after tinyurl.com launched.

My "this" was /*specifically*/ referring to Google's shortner, or at 
least the recent STRONG uptick in the abuse of that shortner. I was 
already well aware of other similar things from other shortners. But 
"this" wasn't referring to them. You infused thoughts/meaning into my 
writing that really wasn't there. (be careful about assuming...) Also, 
when you see my report further down, you'll understand why those others 
are not nearly as much of a concern to me at the moment.

>> Keep in mind that, if a marketer is doing things the right way, they should have no need to obfuscate their own domain name. They should instead proudly use it and not feel the need to hide behind Google's shortner.
> No, that is not at all true. The primary use of a shorter is to shorten a long URL to something that someone can type in.

I've acknowledged that there are some good reasons for a shortner - but 
the vast majority of the times I'm seeing them - in both ham and spam - 
that is NOT the case! The are shortening things like average-sized 
domains with either no directory, or with a short directory or page 
names after the domain. This is the VAST MAJORITY of the shortners I'm 
seeing in both hams and spams.

> Clicking a URL in an email is the height of stupidity, so having a short URL that someone can realistically type into a browser is much better.

If I spent just a little more time on this, I could collect a large 
number of Google shortner URLs that are malicious - where my 
malwarebytes is blocking access to the page to which it is trying to 
redirect. And these are still "live"! Do you really think that more than 
a tiny percent of those who saw those in their mailbox (both legit and 
spams) are manually typing in the URL and not just clicking on it? And 
in that exceedingly rare occasion where somebody types in the URL and it 
redirects to a malicious page that tries to install a virus, are they 
ANY better off than having just clicked on it? Even the best point I can 
think of that you might have had - that this might help them to better 
recognize a phishing URL for example - is lost since BOTH the phish and 
their bank's web site is going to be indistinguishable until AFTER 
they've launched the shortner (whether by clicking or typing). I think 
you just mistakenly bolstered my argument against this over-usage (and 
often inappropriate usage) of shortners!

>> THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner, and we're amplifying other "bad" points.
> Well, at least you are warning people. However, what you are doing is, 
> frankly, dumb; if you think there's a huge problem, you can simply 
> check the target URLs. 

That incurs a significant amount of extra resources for DNSBLs and spam 
filters - and such automated lookups could also put a huge extra burden 
on Google's servers - and who knows at this point if this is even 
reliable - Google might easily start putting captchas in the way or 
otherwise consider such lookups to be abusive and/or mistake them for 
malicious bots... I'm definitely going to pursue this further - but 
wow... that you would suggest this... I think spam puts ENOUGH burden on 
spam filters and mail system as it is!

>> Yes, there are many legitimate uses of Google's shortner, too. However, we are now at a point where a VERY large % (a majority?) of uses of these headed to a typical user's mailbox are egregious spams, and a significant additional portion are likely-spams.
> Any evidence of this?


EVIDENCE/STATS:

I ran stats on a sample set of a few thousand mailboxes, over a period 
of several hours today (mostly during business hours for these 
particular organizations who use these mailboxes) - and this produced a 
combined 24K legit messages, and 5K spams (I'm guessing that most 
systems have more spams per amount of hams? But those were the numbers 
for this server.)

-----------------------------------
NOTE: The sum of individual shortner-hits totals below can be LARGER 
than the total messages that had hits on ANY shortner - Why? - Because 
in a few cases, the same message can have hits on more than one of these 
shortners
-----------------------------------

I SEARCHED EACH HAM AND SPAM CORPUS FOR MANY OF HUNDREDS OF URL SHORTNERS

HERE ARE THE RESULTS:

-----------------------------------
STATS FROM SPAM:
286 total spams blocked that had a shortner, out of hundreds of URL 
shortners I had searched on
(<10 that *MIGHT* be FPs - they were definitely questionable at best - 
btw, zero of those questionable ones led to ANY kind of invaluement 
blacklisting, even with the adjustments I described)

of that 286 SPAMS...
262 used goo.gl
24 used bit.ly
1 used eepurl.com
0 used tinyurl.com
0 used t.co
ONLY 17 used one of ALL of the others COMBINED

-----------------------------------
STATS FROM HAM:

187 total legit messages had a hit on at least one of hundreds of URL 
shortners I had searched on
(a dozen or two of those were "false negatives", most of which used 
goo.gl - and that level of missed spams was much higher prior to the 
changes I had recently made regarding Google's shortner.)

of that 187
83 used goo.gl
59 used bit.ly
40 used eepurl.com
3 used tinyurl.com
53 used t.co
42 for ALL of the others COMBINED

-----------------------------------

NOTE: eepurl.com was under my radar - but since they predominantly hit 
in hams and had extreme few spam hits (just 1), this one is probably 
well run.

ALSO: I initially accidentally left out "t.co" in the overall stats, but 
they only had 53 ham hits, and zero spam hits - so it didn't seem worth 
going back and updating the overall totals to include those messages 
that weren't already in those totals.

Clearly, the results here show that, at the moment - Google's shortner 
is DOMINATING in its spam usage, where 92% (262 of 286) of ALL spam that 
contained shortners used Google. All the rest shared the remaining 8%, 
with bit.ly being in the distinct 2nd place.

I should note that I already had SIMILAR rules in place (for YEARS) that 
caused higher spam scoring for bit.ly (and a few others) - and I think 
that is *one* of the reasons why I'm not picking on them as much at the 
present time. (besides the fact that they are not used for spamming 
nearly as much at the moment as Google's shortner)

So why am I saying this about Google now - but didn't do a similar post 
about bit.ly or TinyURL or others way back when?

(1) the lines never seemed quite as blurred, or at least blurred so 
quickly, as they are now with Google. The bit.ly and tinyurl spam usage 
seemed more limited in scope, and rules that would hit on it them 
without hitting on legit mail usage of those shortners were easier to 
create. But part of the problem here is that the recent spam usage of 
goo.gl is getting a little bit more sophisticated in their usage of 
tactics to evade filters - making the writing of those rules a little 
harder. Having said that - this particular analysis has alerted me to 
the fact that I'm now overdue to tighten up on bit.ly, too - since some 
of these same things are happening more and more with bit.ly - I just 
didn't notice it as much since LARGER problems with goo.gl are hogging 
my attention.

HOWEVER...

(2) Even so - I'm seeing a distinct pattern whereby bit.ly is QUICK to 
dead-end shortners created by spammers... while Google is SLOW!!! And 
this is EXACTLY why spammers are currently MUST MORE OFTEN choosing to 
use Google's shortner right now. FOR EXAMPLE: For those stats above - I 
went back and checked on the 5 oldest spams in the corpus that used 
bit.ly shortners - and the 5 oldest spams in that corpus from today that 
used goo.gl shortners found in egregious spam. btw - *ALL* of those 5 
goo.gl shortner examples were at least 2 hours OLDER than my oldest 
bit.ly samples - so Google had a two hour "head start". But all were 
about 7-9 hours old, fwiw, when I checked back on those links.

RESULTS:

All 5 of my oldest bit.ly shortners had been disabled :)

All 5 of my oldest goo.gl shortners were fully operational (even if the 
redirect was blocked by my malwarebytes... but Google was TRYING to 
deliver the spam payloads!) :(

And this is consistent with my recent experiences with these goo.gl 
shortners. I've reported several to the Google shortner page as spam 
over the past several weeks - and then I'd stop checking on them after a 
few days passed and they were STILL operational.

(3) And then another problem with goo.gl is that it seems to be getting 
more "institutionalized" with Google's software (etc?). Want a google 
map link? Or a link to Google Drive? I'm pretty sure their software will 
serve up a shortener version - and for good reason. Some of THOSE links 
can be complex and benefit greatly from being shortened. But this ends 
of being a "human shield" for spammers if they can't keep up with 
policing the abuse.

(4) And as I had mentioned, I'm seeing evidence that gray-hat spammers 
and ESPs are jumping on this bandwagon, even MORE so than they ever did 
with other shortners - and I have a HUGE HUGE HUGE PROBLEM WITH THAT. It 
is a loophole where spammer can evade a spam filter and DNSBL's ability 
to FULLY evaluate the *identity* and *reputation* of the sender and the 
sender's message - as such senders then hide behind the shortner. This 
is OK if this is an innocent and necessary link to Google maps, for 
example. This is NOT ok if an ESP says to its clients, "hey guys, we 
have this great new trick where you don't ever have to worry about your 
domain getting blacklisted and more of your messages will now get 
delivered" (more legit senders don't need that help!) And there are 
various shades of gray to work out in between. Also, when I was 
examining the more legit messages that had this shortner - while I did 
find a few example to things like Google maps - I ALSO saw a few 
examples where people were putting that as a link in their signature - 
for web site addresses which were NOT very long in the first place. That 
makes no sense - and makes me wonder, what software is stupidly 
auto-generating those links in the signature for them? (more awful human 
shields!)

(5) Finally - this is Google!!!! - they carry more *instant* legitimacy 
- making them more of a desirable target for spammer to exploit (that is 
- if/when the abuse isn't properly policed, as is happening now) - 
Google is much more desired than just using some other company that 
created a shortner.

I think that spammers and blackhat/grayhat ESPS... are ENJOYING my 
pain!  And I think they LOVE this loophole. Please don't "carry the 
water" for them.

-- 
Rob McEwen
https://www.invaluement.com

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Benny Pedersen <me...@junc.eu>]

Dianne Skoll skrev den 2018-02-21 16:37:

> We do a HEAD request and it works on most URL shorteners.
> 
> The concern voiced in another email about overloading Google's
> infrastructure is quite charming and quaint.

+1

some with icla could add this to spamasssassin with

https://github.com/smfreegard/DecodeShortURLs

so if yes problem is solved imho, but should we consider all _URIHOSTS_ 
as shortners ?

unless thay are confirmed not to do this

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 11:44 AM, Dianne Skoll wrote:
> On Wed, 21 Feb 2018 16:35:27 +0000
> Karol Augustin <ka...@augustin.pl> wrote:
>> I think the point here might be that if Google acted promptly on abuse
>> spammers would stop using shorteners.
> True, that might happen.  OTOH, I see about as many spams with bit.ly
> shorteners as goo.gl shorteners which is not what one might expect if
> bit.ly were really that much more proactive than goo.gl.

I'm sure mileage may vary - but I'm seeing about 10X the abuse for 
goo.gl right now as I see from bit.ly. Also, when I do random checks on 
a handful of abused bitly shortners, a high percentage of them are 
already terminated. But when I do random checks of abused goo.gl 
redirectors, most of them are still operational. (I'm referring to 
redirectors found in spams within the previous few days of when I 
checked them, with at least hours having gone by since the message was 
sent - I know that sounds anecdotal - but as I've been researching this 
in the past weeks, that pattern keeps happening). One thing that could 
potentially make those numbers different - is when you compare one 
system that blocks MUCH spam at the perimeter based on the sending IP, 
such as blocking all Zen-listed spams before DATA.... while another 
system might capture ALL messages and process them all. The latter is 
what my system does. That also might explain the difference in stats?

-- 
Rob McEwen
https://www.invaluement.com

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 16:35:27 +0000
Karol Augustin <ka...@augustin.pl> wrote:

> I think the point here might be that if Google acted promptly on abuse
> spammers would stop using shorteners.

True, that might happen.  OTOH, I see about as many spams with bit.ly
shorteners as goo.gl shorteners which is not what one might expect if
bit.ly were really that much more proactive than goo.gl.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Karol Augustin <ka...@augustin.pl>]

On 2018-02-21 16:31, Dianne Skoll wrote:
> On Wed, 21 Feb 2018 11:29:00 -0500
> Rob McEwen <ro...@invaluement.com> wrote:
> 
>> Nevertheless, it is a shame to have to shift more of the burden onto
>> spam filters to do more work (some of which requires MORE latency) -
>> in order to partly mitigate Google's failure to prevent/correct the
>> abuse.
> 
> Yes, I agree.  On the other hand, IMO a spam filter should block messages
> that point to a taken-down shortened URL.  Although such messages may
> not be downright dangerous, they are still annoying and are still spam.
> There's no way of avoiding the work.
> 

I think the point here might be that if Google acted promptly on abuse
spammers would stop using shorteners.


Karol

--  
Karol Augustin
karol@augustin.pl
http://karolaugustin.pl/
+353 85 775 5312

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 11:29:00 -0500
Rob McEwen <ro...@invaluement.com> wrote:

> Nevertheless, it is a shame to have to shift more of the burden onto 
> spam filters to do more work (some of which requires MORE latency) -
> in order to partly mitigate Google's failure to prevent/correct the
> abuse.

Yes, I agree.  On the other hand, IMO a spam filter should block messages
that point to a taken-down shortened URL.  Although such messages may
not be downright dangerous, they are still annoying and are still spam.
There's no way of avoiding the work.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 11:11 AM, Dianne Skoll wrote:
> I guess I misinterpreted: "...such automated lookups could also put a
> huge extra burden on Google's servers..." from Message-Id
> <b2...@invaluement.com>

Oh yeah, I'd forgotten about that part. it was a more minor point. But 
as I think back on my thought processes at the time I typed those words 
- I was envisioning what would happen if ALL ISPs and hosters and spam 
filtering vendors, SA installations etc...  ALL started doing all of 
those lookups. But yeah, maybe that would still be a drop in the bucket 
compared to all that Google does.

Nevertheless, it is a shame to have to shift more of the burden onto 
spam filters to do more work (some of which requires MORE latency) - in 
order to partly mitigate Google's failure to prevent/correct the abuse. 
In contrast, at this time, bit.ly is running circles around Google in 
terms of quickly shutting down their abused redirectors. I know this 
isn't easy, but there is definitely room for improvement.

But my larger point in that overall post you quoted from, was my concern 
about one organization doing high volume lookups from a single server 
getting blocked or captcha'd.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 10:58:17 -0500
Rob McEwen <ro...@invaluement.com> wrote:

> On 2/21/2018 10:37 AM, Dianne Skoll wrote:
> > The concern voiced in another email about overloading Google's
> > infrastructure is quite charming and quaint.  

> My concern was NEVER about overloading google.

I guess I misinterpreted: "...such automated lookups could also put a
huge extra burden on Google's servers..." from Message-Id
<b2...@invaluement.com>

> My concern was about Google auto-blocking or throwing a captcha at
> very high volume and automated lookups. That is a HUGE difference.

That's a concern, but you can mitigate it slightly by caching the
results of URL expansion.  Also, I suspect most URL-shorteners
expect to be bombarded with requests and are engineered accordingly.
I have seen a CAPTCHA thrown up from the search page, but I've never
seen a goo.gl HEAD request reply with anything other than 404 or 301.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 10:37 AM, Dianne Skoll wrote:
> The concern voiced in another email about overloading Google's
> infrastructure is quite charming and quaint.


My concern was NEVER about overloading google. My concern was about 
Google auto-blocking or throwing a captcha at very high volume and 
automated lookups. That is a HUGE difference.

-- 
Rob McEwen
https://www.invaluement.com

Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

[Dianne Skoll <df...@roaringpenguin.com>]

On Tue, 20 Feb 2018 23:38:53 -0700
"@lbutlr" <kr...@kreme.com> wrote:

> As I suspected, it is possible to get the goo.gl target URL without
> loading the site, though using curl is probably not realistic in this
> specific case.

We do a HEAD request and it works on most URL shorteners.

The concern voiced in another email about overloading Google's
infrastructure is quite charming and quaint.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-21 (00:52 MST), Charles Sprickman <sp...@bway.net> wrote:
> 
> You can also see all the analytics by appending “.info” to the URL, eg: http://goo.gl/ylUAd.info

True, but that is a web browser solution, not something that could, for example, be scripted (well, not easily or realistically for this sort of task).

Also, appending a "+" I believe does the same thing, and is easier to type. http://goo.gl/ylUAd+

Either way, that redirects to a URL in the form

<https://goo.gl/#analytics/goo.gl/<shortcode>/all_time>

So that could probably be scraped as well if you really wanted to scrape a web page.

Still, this isn't a viable solution for spam checking, though I suspect there is a low-cost "get redirect URI" library that would have less overhead than curl and could grab the Location from the 301 (or 302?) redirect, much like curl can do with a -I or -i flags.

Regardless, Reindl was clearly wrong that checking the shortner targets would require loading the target URL.

Maybe curl ain't so bad?

# time curl -sI http://goo.gl/ylUAd | grep -o "http.*"
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

real    0m0.098s
user    0m0.000s
sys     0m0.008s

I threw this together in a few minutes and it seems to work:

---
#!/bin/bash

  if [ "$1" = "" ]; then
   echo "usage $0 <URL>"
   exit 1
  else 
   URL="$1"
  fi

myInfo=$(curl -sI $URL | grep -o "http.*")
if [ "$myInfo" = "" ]; then
   exit 404
else
   echo $myInfo
fi
---

 # curlri www.covisp.net # This uses rewrite
https://www.covisp.net/
 # curlri www.google.com # No redirect, so returned 404
 # curlri http://goo.gl/ylUAd # Voila!
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

It even works for the t.co link:

 # curlri http://t.co/UShCXQG
http://www.youtube.com/watch?v=dfJ6oCdjY9Y

-- 
It is one thing to be mistaken; it is quite another to be willfully
ignorant ~Cecil Adams

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Charles Sprickman <sp...@bway.net>]

> On Feb 21, 2018, at 1:38 AM, @lbutlr <kr...@kreme.com> wrote:
> 
> On 2018-02-20 (22:10 MST), Reindl Harald <h....@thelounge.net> wrote:
>> 
>> you may hit confirmation-urls (both ham and spam), trigger actions, trigger *one-time* urls which are invalid for the user after a dumb bot used them not talking about that it would be illegal in many countries in case of private ham-mails
> 
> As I suspected, it is possible to get the goo.gl target URL without loading the site, though using curl is probably not realistic in this specific case.
> 
> $ curl -s "http://goo.gl/ylUAd" | grep -o "http[^\"]*"
> http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135 <http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135>

You can also see all the analytics by appending “.info” to the URL, eg: http://goo.gl/ylUAd <http://goo.gl/ylUAd>.info

Charles

> 
> $ curl -s "http://bit.ly/savecastle" | grep -o "http[^\"]*"
> http://community.livejournal.com/castle_tv/28872.html
> 
> Doesn't work with t.co, but that is not surprising since twitter uses that specifically to hide URLs, considering them all their property that must go through their servers.
> 
> -- 
> Mos Eisley spaceport. You will never find a more wretched hive of scum
> and villainy. We must be cautious.
> 

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-21 (09:27 MST), Alex <my...@gmail.com> wrote:
> 
> This is what DecodeShortURLs is for
> https://github.com/smfreegard/DecodeShortURLs

Aha! I knew something like that must exist! 

-- 
EIR OWN DESTINY. THEY TOUCH THE EARTH LIGHTLY.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Alex <my...@gmail.com>]

On Wed, Feb 21, 2018 at 1:38 AM, @lbutlr <kr...@kreme.com> wrote:
> On 2018-02-20 (22:10 MST), Reindl Harald <h....@thelounge.net> wrote:
>>
>> you may hit confirmation-urls (both ham and spam), trigger actions, trigger *one-time* urls which are invalid for the user after a dumb bot used them not talking about that it would be illegal in many countries in case of private ham-mails
>
> As I suspected, it is possible to get the goo.gl target URL without loading the site, though using curl is probably not realistic in this specific case.
>
> $ curl -s "http://goo.gl/ylUAd" | grep -o "http[^\"]*"
> http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135
>
> $ curl -s "http://bit.ly/savecastle" | grep -o "http[^\"]*"
> http://community.livejournal.com/castle_tv/28872.html
>
> Doesn't work with t.co, but that is not surprising since twitter uses that specifically to hide URLs, considering them all their property that must go through their servers.

This is what DecodeShortURLs is for
https://github.com/smfreegard/DecodeShortURLs

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[David Jones <dj...@ena.com>]

On 03/18/2018 06:59 PM, RW wrote:
> On Sun, 18 Mar 2018 18:24:36 -0500
> David Jones wrote:
> 
>> On 03/18/2018 06:01 PM, Alan Hodgson wrote:
>>> On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:
>>>> I have Steve Freegard's DecodeShortURLs.pm installed but didn't
>>>> get any HAS_SHORT_URL hits on this one:
> 
> 
>>> Is it getting any hits? It definitely hits on that one in a test
>>> here.
>>>
>>> Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to
>>> work at all.
> 
> I don't think you actually need DBD::SQLite, it just enables caching to
> work.
> 
>>>    
>>
>> I am getting other HAS_SHORT_URL hits.
> 
> Probably they are http links and you are missing  LWP::Protocol::https
> which is needed to handle the https link in that email.
> 
> 
> 
> 

I was able to get LWP::Protocol::https installed on my mail filters and 
now I am getting HAS_SHORT_URL hits on https links.  Thanks for the tip!

-- 
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[RW <rw...@googlemail.com>]

On Sun, 18 Mar 2018 18:24:36 -0500
David Jones wrote:

> On 03/18/2018 06:01 PM, Alan Hodgson wrote:
> > On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:  
> >> I have Steve Freegard's DecodeShortURLs.pm installed but didn't
> >> get any HAS_SHORT_URL hits on this one:


> > Is it getting any hits? It definitely hits on that one in a test
> > here.
> > 
> > Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to
> > work at all.

I don't think you actually need DBD::SQLite, it just enables caching to
work.

> >   
> 
> I am getting other HAS_SHORT_URL hits. 

Probably they are http links and you are missing  LWP::Protocol::https
which is needed to handle the https link in that email.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[David Jones <dj...@ena.com>]

On 03/18/2018 06:01 PM, Alan Hodgson wrote:
> On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:
>> I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any
>> HAS_SHORT_URL hits on this one:
>>
>> https://pastebin.com/t85b0Bns <https://pastebin.com/t85b0Bns >
> 
> Is it getting any hits? It definitely hits on that one in a test here.
> 
> Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to work at 
> all.
> 

I am getting other HAS_SHORT_URL hits.  My 8 SA boxes are setup 
identically with those 2 perl modules.  The original server that 
received this message does have other hits on HAS_SHORT_URL but this 
message still is not hitting.  Now it's scoring a 10.4 and would be 
blocked because of DCC_CHECK and other things have caught up to it.

-- 
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Alan Hodgson <ah...@lists.simkin.ca>]

On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:
> 
I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any 
> HAS_SHORT_URL hits on this one:
> 
> https://pastebin.com/t85b0Bns


Is it getting any hits? It definitely hits on that one in a test here.

Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to work
at all.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[David Jones <dj...@ena.com>]

On 03/10/2018 10:57 AM, Rob McEwen wrote:
> On 3/10/2018 11:43 AM, Matus UHLAR - fantomas wrote:
>>> On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
>>>> this is apparently not the case of one url redirector (shortener) 
>>>> points to
>>>> another shortener.
>>>>
>>>> I really hope that the DecodeShortURLs only checks fopr redirection 
>>>> at those
>>>> known redirectors (shorteners), not each http->https shortener and only
>>>> evaluates redirection between them, ignoring http->https redirects
>>
>> On 10.03.18 11:32, Rob McEwen wrote:
>>> But also keep in mind that it is NOT rare for the initial shortner 
>>> found in a spam... to redirect to a spammer's page (that isn't a URL 
>>> shortner), then THAT page then redirects to the spammer's OTHER page 
>>> (that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
>>> very many places... but the second page (that is often the final 
>>> destination)... *is* heavily blacklisted. Often, the first page is 
>>> hosted at a legit site that was hacked into by criminal spammers - 
>>> and is HARDER for blacklists to list due to their good reputation. 
>>
>> isn't this thread and the plugin discussed about shorteners?
>> I am aware of other possibilities to do redirects and about 
>> consequencies of
>> following them.
>> note that for following the redirect, the destination must be 
>> configured via
>> url_shortener directive.
>> what I wonder is, if there's real value in following the redirects.
>> (and if the following stops when the URL is blacklisted, since we're done
>> here).
>>
> 
> The URL shortner follower plugin that Steve Freegard wrote... ALREADY 
> follows my suggestions about following non-shortner redirects (after the 
> initial shortner redirect). (except I'm not sure it already does my 
> https suggestion?) ...and for good reason. You're tremendously 
> undervaluing or just not understanding my last post. I suggestion you 
> re-read it a couple of times. This tactic I mentioned that spammers 
> use... is COMMON, and not following my last suggestion opens up a big 
> loophole for spammers. Such spammers would be delighted by your last 
> comment... and Steve Freegard went a different route for good reason. 
> Keep in mind, I haven't veered off-topic because we're STILL talking 
> about a possible chain of redirects that was still STARTED by a URL 
> shortner.
> 

I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any 
HAS_SHORT_URL hits on this one:

https://pastebin.com/t85b0Bns

I don't see much that can be done other than training Bayes which will 
probably not help the next variation of this junk.  I have reported to 
Spamcop and Microsoft -- they MUST do a better job of outbound mail 
filtering!

-- 
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 3/10/2018 11:43 AM, Matus UHLAR - fantomas wrote:
>> On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
>>> this is apparently not the case of one url redirector (shortener) 
>>> points to
>>> another shortener.
>>>
>>> I really hope that the DecodeShortURLs only checks fopr redirection 
>>> at those
>>> known redirectors (shorteners), not each http->https shortener and only
>>> evaluates redirection between them, ignoring http->https redirects
>
> On 10.03.18 11:32, Rob McEwen wrote:
>> But also keep in mind that it is NOT rare for the initial shortner 
>> found in a spam... to redirect to a spammer's page (that isn't a URL 
>> shortner), then THAT page then redirects to the spammer's OTHER page 
>> (that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
>> very many places... but the second page (that is often the final 
>> destination)... *is* heavily blacklisted. Often, the first page is 
>> hosted at a legit site that was hacked into by criminal spammers - 
>> and is HARDER for blacklists to list due to their good reputation. 
>
> isn't this thread and the plugin discussed about shorteners?
> I am aware of other possibilities to do redirects and about 
> consequencies of
> following them.
> note that for following the redirect, the destination must be 
> configured via
> url_shortener directive.
> what I wonder is, if there's real value in following the redirects.
> (and if the following stops when the URL is blacklisted, since we're done
> here).
>

The URL shortner follower plugin that Steve Freegard wrote... ALREADY 
follows my suggestions about following non-shortner redirects (after the 
initial shortner redirect). (except I'm not sure it already does my 
https suggestion?) ...and for good reason. You're tremendously 
undervaluing or just not understanding my last post. I suggestion you 
re-read it a couple of times. This tactic I mentioned that spammers 
use... is COMMON, and not following my last suggestion opens up a big 
loophole for spammers. Such spammers would be delighted by your last 
comment... and Steve Freegard went a different route for good reason. 
Keep in mind, I haven't veered off-topic because we're STILL talking 
about a possible chain of redirects that was still STARTED by a URL 
shortner.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Matus UHLAR - fantomas <uh...@fantomas.sk>]

>On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
>>this is apparently not the case of one url redirector (shortener) 
>>points to
>>another shortener.
>>
>>I really hope that the DecodeShortURLs only checks fopr redirection 
>>at those
>>known redirectors (shorteners), not each http->https shortener and only
>>evaluates redirection between them, ignoring http->https redirects

On 10.03.18 11:32, Rob McEwen wrote:
>But also keep in mind that it is NOT rare for the initial shortner 
>found in a spam... to redirect to a spammer's page (that isn't a URL 
>shortner), then THAT page then redirects to the spammer's OTHER page 
>(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
>very many places... but the second page (that is often the final 
>destination)... *is* heavily blacklisted. Often, the first page is 
>hosted at a legit site that was hacked into by criminal spammers - 
>and is HARDER for blacklists to list due to their good reputation. 

isn't this thread and the plugin discussed about shorteners?
I am aware of other possibilities to do redirects and about consequencies of
following them.
note that for following the redirect, the destination must be configured via
url_shortener directive. 

what I wonder is, if there's real value in following the redirects.
(and if the following stops when the URL is blacklisted, since we're done
here).

-- 
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Posli tento mail 100 svojim znamim - nech vidia aky si idiot
Send this email to 100 your friends - let them see what an idiot you are

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
> this is apparently not the case of one url redirector (shortener) 
> points to
> another shortener.
>
> I really hope that the DecodeShortURLs only checks fopr redirection at 
> those
> known redirectors (shorteners), not each http->https shortener and only
> evaluates redirection between them, ignoring http->https redirects 


But also keep in mind that it is NOT rare for the initial shortner found 
in a spam... to redirect to a spammer's page (that isn't a URL 
shortner), then THAT page then redirects to the spammer's OTHER page 
(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
very many places... but the second page (that is often the final 
destination)... *is* heavily blacklisted. Often, the first page is 
hosted at a legit site that was hacked into by criminal spammers - and 
is HARDER for blacklists to list due to their good reputation. Then the 
2nd final destination page is just a heavily blacklisted spammer's 
throwaway domain. Therefore, there is some value to following the 
redirects a little further than what you suggest, and then collecting 
all of those host names or domains, checking ALL of them against 
URI/domain blacklists. (within reason... after too many redirects, it is 
better to just stop and add points to the spam score)

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Matus UHLAR - fantomas <uh...@fantomas.sk>]

>On 3/10/2018 3:20 AM, Matus UHLAR - fantomas wrote:
>>do you have an example of any chained redirection not suspicious?

On 10.03.18 11:04, Rob McEwen wrote:
>I haven't examined the code for that plugin very much (yet!).... but 
>one type of very common redirect that is very innocent... is the fact 
>that a MASSIVE percentage of web sites have switched to all SSL sites 
>in the past few years, due to Google now punishing http and/or 
>rewarding https... in the search engine rankings. But there are still 
>many links and redirectors pointing to the non-SSL versions of those 
>sites, which is then typically redirected to the SSL version. 

this is apparently not the case of one url redirector (shortener) points to
another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection at those
known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects

-- 
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 3/10/2018 3:20 AM, Matus UHLAR - fantomas wrote:
> do you have an example of any chained redirection not suspicious? 


I haven't examined the code for that plugin very much (yet!).... but one 
type of very common redirect that is very innocent... is the fact that a 
MASSIVE percentage of web sites have switched to all SSL sites in the 
past few years, due to Google now punishing http and/or rewarding 
https... in the search engine rankings. But there are still many links 
and redirectors pointing to the non-SSL versions of those sites, which 
is then typically redirected to the SSL version. Therefore, if the code 
for this plugin (and others using this tactic) doesn't do this 
already... it should probably not count THAT particular redirect as a 
spam indicator, when counting the total number of redirects.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Matus UHLAR - fantomas <uh...@fantomas.sk>]

On 07.03.18 10:59, shanew@shanew.net wrote:
>Just FYI, it does add 3.0 points as soon as it sees any chaining at
>all.  The other 5.0 points get added at 10 redirections.  That said,
>I think you're guess is right that redirections start to look really
>suspicious after just 3 or 4.

do you have an example of any chained redirection not suspicious?

>On Sat, 3 Mar 2018, @lbutlr wrote:
>
>>On Feb 26, 2018, at 09:55, shanew@shanew.net wrote:
>>>
>>>This is why the DecodeShortURLs plugin has an explicit limit of 10
>>>lookups (and penalizes such with a total of 8 points).
>>
>>I’d guess more than one redirect is highly suspicious and more than two is probably a waste of time, just score 5.0 and be done with it.
>>
>>Has anyone done any analysis on multi-redirects?

-- 
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
How does cat play with mouse? cat /dev/mouse

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Matthias Leisi <ma...@leisi.net>]

> Hi Guys! We provide an URIBL that already have a script in Perl to expand
> redirections until no more redirections:

I would be uneasy to follow such redirections on a production email system (as opposed to eg a spamtrap system). You are likely „confirming“ live email addresses to the spammer by blindly following links and redirects, which may be tied to individual email addresses.

— Matthias

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-02-26 10:41 GMT-03:00 Dianne Skoll <df...@roaringpenguin.com>:

> On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
> David B Funk <db...@engineering.uiowa.edu> wrote:
>
> > So my bet is that the spammers are crafty enough to check things like
> > browser referrer, cookies, etc to detect/differentiate a browser vs a
> > link-checker.
>
> Yep.  You need to fake your User-Agent (not hard) and put a limit on
> the number of redirections you follow or you can end up in trouble.
>
> I picked a likely-looking agent out of our logs and use this:
>
> $ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64;
> Trident/7.0; SLCC2;');
>
> Might need revising later because it's pretty ancient!
>


That is right Dianne!

Our script is using "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0)
Gecko/20100101 Firefox/57.0":

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0

This user agent was most used, some time ago, according to this site:

https://techblog.willshouse.com/2012/01/03/most-common-user-agents/

This is out of date now, than you can update it if you want. We will do it
soon.



>
> Regards,
>
> Dianne.
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
David B Funk <db...@engineering.uiowa.edu> wrote:

> So my bet is that the spammers are crafty enough to check things like
> browser referrer, cookies, etc to detect/differentiate a browser vs a
> link-checker.

Yep.  You need to fake your User-Agent (not hard) and put a limit on
the number of redirections you follow or you can end up in trouble.

I picked a likely-looking agent out of our logs and use this:

$ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2;');

Might need revising later because it's pretty ancient!

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-02-26 3:07 GMT-03:00 David B Funk <db...@engineering.uiowa.edu>:

>
> Just be careful how you do that "expand redirections until no more
> redirections" or you may get caught in a spammer trap.
>
> If you're going thru a professional redirect site like goo.gl or bit.ly
> you're probably pretty safe but if it's a dedicated spammer site be ware.
>
> I was testing some redirection expantions on URLs from spam and found a
> site that clearly had been crafted to foil this kind of thing.
>
> It was in one of those "check this out" spams which contains one line of
> greeting and then a URL.
>
> When I grabbed it using curl it returned a 301 redirect, so I grabbed that
> target, which lead to another 301, lather-rinse-repeat ad nausium.
> However if you used a browser it went to the target "burn fat pills" site
> in just two redirects.
>
> So my bet is that the spammers are crafty enough to check things like
> browser referrer, cookies, etc to detect/differentiate a browser vs a
> link-checker.
>
>
Great! If the spammer uses same redirection URLs, as a loop, the script
will ignore redirection on first recurrence.

But this script is a beta and it is not is not mature enough to deal with
loop redirects of distinct URLs. Maybe add a count limit resolves this issue,
like 10 times max redirections.

Most likely I will implement this, but my need is for someone to help me
implement this script inside the SA, so that some users here in Brazil can
use our URIBL easily.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[sh...@shanew.net]

Just FYI, it does add 3.0 points as soon as it sees any chaining at
all.  The other 5.0 points get added at 10 redirections.  That said,
I think you're guess is right that redirections start to look really
suspicious after just 3 or 4.


On Sat, 3 Mar 2018, @lbutlr wrote:

> On Feb 26, 2018, at 09:55, shanew@shanew.net wrote:
>>
>> This is why the DecodeShortURLs plugin has an explicit limit of 10
>> lookups (and penalizes such with a total of 8 points).
>
> I’d guess more than one redirect is highly suspicious and more than two is probably a waste of time, just score 5.0 and be done with it.
>
> Has anyone done any analysis on multi-redirects?
>
>

-- 
Public key #7BBC68D9 at            |                 Shane Williams
http://pgp.mit.edu/                |      System Admin - UT CompSci
=----------------------------------+-------------------------------
All syllogisms contain three lines |              shanew@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Benny Pedersen <me...@junc.eu>]

John Hardin skrev den 2018-03-03 19:28:

>>> This is why the DecodeShortURLs plugin has an explicit limit of 10
>>> lookups (and penalizes such with a total of 8 points).
>> I’d guess more than one redirect is highly suspicious and more than 
>> two is probably a waste of time, just score 5.0 and be done with it.
> +1

add blacklist internaly to DecodeShortURLs plugin, and reduce redirector 
list to who support abuse reports only

bit.ly is safe to test

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[John Hardin <jh...@impsec.org>]

On Sat, 3 Mar 2018, @lbutlr wrote:

> On Feb 26, 2018, at 09:55, shanew@shanew.net wrote:
>>
>> This is why the DecodeShortURLs plugin has an explicit limit of 10
>> lookups (and penalizes such with a total of 8 points).
>
> I’d guess more than one redirect is highly suspicious and more than two is probably a waste of time, just score 5.0 and be done with it.

+1

-- 
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@impsec.org    FALaholic #11174     pgpk -a jhardin@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
   Back in 1969 the technology to fake a Moon landing didn't exist,
   but the technology to actually land there did.
   Today, it is the opposite.                               -- unknown
-----------------------------------------------------------------------
  10 days until Albert Einstein's 139th Birthday

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On Feb 26, 2018, at 09:55, shanew@shanew.net wrote:
> 
> This is why the DecodeShortURLs plugin has an explicit limit of 10
> lookups (and penalizes such with a total of 8 points).

I’d guess more than one redirect is highly suspicious and more than two is probably a waste of time, just score 5.0 and be done with it. 

Has anyone done any analysis on multi-redirects?

-- 
This is my signature. There are many like it, but this one is mine.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Amir Caspi <ce...@3phase.com>]

> On Feb 26, 2018, at 11:00 AM, Kevin A. McGrail <ke...@mcgrail.com> wrote:
> 
>> DecodeShortURLs has been on my list of must-have plugins for years, so 
>> I was a little surprised it took so long for someone to mention it 
>> in this thread.
> Yeah, my firm is going to look at subsidizing it's addition to SA and Karsten agreed.  Just trying to get the time.
> 
FWIW, I just implemented this plugin last week after this email thread, and it was trivial to include on my (old, very EOL) system.  Hopefully including it in the default SA distribution is not too difficult.

(I also finally got DKIM validation working on my virtual hosting system, including with the MailScanner setup. Finally, auth rules are working as they should!)

Thanks!

--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-02-27 9:03 GMT-03:00 Rob McEwen <ro...@invaluement.com>:

> On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:
>
> DecodeShortURLs has been on my list of must-have plugins for years, so
> I was a little surprised it took so long for someone to mention it
> in this thread.
>
> Yeah, my firm is going to look at subsidizing it's addition to SA and
> Karsten agreed.  Just trying to get the time.
>
>
> Kevin,
>
> Excellent! Also... for those situations where there are multiple cascading
> redirecting websites (after the initial URL shortners link found in the
> message), please consider having an option for *every* non-whitelisted URI
> in the chain to be added to the checks against blacklists. OFTEN - every
> single domain in that chain (past the initial URL shortner) is a
> compromised web site or spammer's website, not just the final destination
> web site.
>

Also, you can put more than one shortener (not only hacked websites) inside
same chain:

http://bit.ly/2GLA8Ss => https://goo.gl/5fnqaC => http://spfbl.net/en/uribl/

Our script can already solve this chain:

root@mx-br:/home/ubuntu# ./uribl.pl "http://bit.ly/2GLA8Ss"
(spfbl.net) is not listed in 'uribl.spfbl.net'.

Kevin. You can use our script as model to your own implementation.


> --
> Rob McEwenhttps://www.invaluement.com
>
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["Kevin A. McGrail" <ke...@mcgrail.com>]

I will add it to the list, thanks. 
Regards,
KAM

On February 27, 2018 7:03:16 AM EST, Rob McEwen <ro...@invaluement.com> wrote:
>On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:
>>> DecodeShortURLs has been on my list of must-have plugins for years,
>so
>>> I was a little surprised it took so long for someone to mention it
>>> in this thread. 
>>
>> Yeah, my firm is going to look at subsidizing it's addition to SA and
>
>> Karsten agreed.  Just trying to get the time.
>>
>
>Kevin,
>
>Excellent! Also... for those situations where there are multiple 
>cascading redirecting websites (after the initial URL shortners link 
>found in the message), please consider having an option for *every* 
>non-whitelisted URI in the chain to be added to the checks against 
>blacklists. OFTEN - every single domain in that chain (past the initial
>
>URL shortner) is a compromised web site or spammer's website, not just 
>the final destination web site.
>
>-- 
>Rob McEwen
>https://www.invaluement.com
>  

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:
>> DecodeShortURLs has been on my list of must-have plugins for years, so
>> I was a little surprised it took so long for someone to mention it
>> in this thread. 
>
> Yeah, my firm is going to look at subsidizing it's addition to SA and 
> Karsten agreed.  Just trying to get the time.
>

Kevin,

Excellent! Also... for those situations where there are multiple 
cascading redirecting websites (after the initial URL shortners link 
found in the message), please consider having an option for *every* 
non-whitelisted URI in the chain to be added to the checks against 
blacklists. OFTEN - every single domain in that chain (past the initial 
URL shortner) is a compromised web site or spammer's website, not just 
the final destination web site.

-- 
Rob McEwen
https://www.invaluement.com
  

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 2/26/2018 12:55 PM, shanew@shanew.net wrote:
> On Mon, 26 Feb 2018, David B Funk wrote:
>
>> Just be careful how you do that "expand redirections until no more 
>> redirections" or you may get caught in a spammer trap.
>
> This is why the DecodeShortURLs plugin has an explicit limit of 10
> lookups (and penalizes such with a total of 8 points).
>
> DecodeShortURLs has been on my list of must-have plugins for years, so
> I was a little surprised it took so long for someone to mention it
> in this thread. 

Yeah, my firm is going to look at subsidizing it's addition to SA and 
Karsten agreed.  Just trying to get the time.

Regards,

KAM

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[sh...@shanew.net]

On Mon, 26 Feb 2018, David B Funk wrote:

> Just be careful how you do that "expand redirections until no more 
> redirections" or you may get caught in a spammer trap.

This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).

DecodeShortURLs has been on my list of must-have plugins for years, so
I was a little surprised it took so long for someone to mention it
in this thread.


-- 
Public key #7BBC68D9 at            |                 Shane Williams
http://pgp.mit.edu/                |      System Admin - UT CompSci
=----------------------------------+-------------------------------
All syllogisms contain three lines |              shanew@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[David B Funk <db...@engineering.uiowa.edu>]

On Sun, 25 Feb 2018, LeandroCarlosRodrigues wrote:

> Amir Caspi wrote
>> On that note -- regardless of what OTHER HW/SW solutions might do, since
>> this is a SpamAssassin mailing list ... is there any facility to implement
>> this in SA?  That is, when calling the URIBL plugin, could it check both
>> the shortened URL and the expanded URL (for known shorteners) ?  Does that
>> facility already exist and I missed it?
>
> Hi Guys! We provide an URIBL that already have a script in Perl to expand
> redirections until no more redirections:
[snip..]

Just be careful how you do that "expand redirections until no more redirections" 
or you may get caught in a spammer trap.

If you're going thru a professional redirect site like goo.gl or bit.ly you're 
probably pretty safe but if it's a dedicated spammer site be ware.

I was testing some redirection expantions on URLs from spam and found a site 
that clearly had been crafted to foil this kind of thing.

It was in one of those "check this out" spams which contains one line of 
greeting and then a URL.

When I grabbed it using curl it returned a 301 redirect, so I grabbed that 
target, which lead to another 301, lather-rinse-repeat ad nausium.
However if you used a browser it went to the target "burn fat pills" site in 
just two redirects.

So my bet is that the spammers are crafty enough to check things like browser 
referrer, cookies, etc to detect/differentiate a browser vs a link-checker.


-- 
Dave Funk                                  University of Iowa
<dbfunk (at) engineering.uiowa.edu>        College of Engineering
319/335-5751   FAX: 319/384-0549           1256 Seamans Center
Sys_admin/Postmaster/cell_admin            Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[RW <rw...@googlemail.com>]

On Sun, 25 Feb 2018 05:25:06 -0700 (MST)
LeandroCarlosRodrigues wrote:

> Amir Caspi wrote
> > On that note -- regardless of what OTHER HW/SW solutions might do,
> > since this is a SpamAssassin mailing list ... is there any facility
> > to implement this in SA?  That is, when calling the URIBL plugin,
> > could it check both the shortened URL and the expanded URL (for
> > known shorteners) ?  Does that facility already exist and I missed
> > it?  
> 
> Hi Guys! We provide an URIBL that already have a script in Perl to
> expand redirections until no more redirections:
> 

There's already a plugin to do that.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[LeandroCarlosRodrigues <le...@spfbl.net>]

Amir Caspi wrote
> On that note -- regardless of what OTHER HW/SW solutions might do, since
> this is a SpamAssassin mailing list ... is there any facility to implement
> this in SA?  That is, when calling the URIBL plugin, could it check both
> the shortened URL and the expanded URL (for known shorteners) ?  Does that
> facility already exist and I missed it?

Hi Guys! We provide an URIBL that already have a script in Perl to expand
redirections until no more redirections:

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0
<https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0>  

Our URIBL ignores the intermediary URLs. The target is the final URL,
that will be viewed by user:

<code>root@mx-br:/home/ubuntu# ./uribl.pl "http://goo.gl/ylUAd"
(www.hollywoodreporter.com) is not listed in 'uribl.spfbl.net'.</code>

More about consumption of our URIBL can be read here:

http://spfbl.net/en/uribl <http://spfbl.net/en/uribl>  

Could you guys help me adjust this script to run inside SA?



--
Sent from: http://spamassassin.1065346.n5.nabble.com/SpamAssassin-Users-f3.html

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Amir Caspi <ce...@3phase.com>]

> On Feb 21, 2018, at 12:45 PM, Dianne Skoll <df...@roaringpenguin.com> wrote:
> 
> Someone earlier posted a link to https://github.com/smfreegard/DecodeShortURLs

Oops, I missed that... must have thought it was just about decoding and not about SA.  Thanks for clarifying!

--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 12:41:05 -0700
Amir Caspi <ce...@3phase.com> wrote:

> On that note -- regardless of what OTHER HW/SW solutions might do,
> since this is a SpamAssassin mailing list ... is there any facility
> to implement this in SA?

Someone earlier posted a link to https://github.com/smfreegard/DecodeShortURLs

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 2:41 PM, Amir Caspi wrote:
>> On Feb 21, 2018, at 9:57 AM, Dianne Skoll <df...@roaringpenguin.com> wrote:
>>
>> That's why you only want to do it for URLs that are
>> absolutely known to be shortened URLs.  You have to keep a list of
>> known URL-shorteners.
> On that note -- regardless of what OTHER HW/SW solutions might do, since this is a SpamAssassin mailing list ... is there any facility to implement this in SA?  That is, when calling the URIBL plugin, could it check both the shortened URL and the expanded URL (for known shorteners) ?  Does that facility already exist and I missed it?

If I'm not confusing things, someone answered things earlier in this 
thread, as follows:

On 2/21/2018 11:27 AM, Alex wrote:
> This is what DecodeShortURLs is for
> https://github.com/smfreegard/DecodeShortURLs
>

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Amir Caspi <ce...@3phase.com>]

> On Feb 21, 2018, at 9:57 AM, Dianne Skoll <df...@roaringpenguin.com> wrote:
> 
> That's why you only want to do it for URLs that are
> absolutely known to be shortened URLs.  You have to keep a list of
> known URL-shorteners.

On that note -- regardless of what OTHER HW/SW solutions might do, since this is a SpamAssassin mailing list ... is there any facility to implement this in SA?  That is, when calling the URIBL plugin, could it check both the shortened URL and the expanded URL (for known shorteners) ?  Does that facility already exist and I missed it?

It seems like this wouldn't be too horrible to implement within the existing URIBL plugin, unless I'm missing something obvious.

Thanks!

						--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 16:48:40 +0000
Anthony Cartmell <aj...@fonant.com> wrote:

> If you mean following URLs in messages, you do need to be aware that
> this can break one-time login links.

Big time.  That's why you only want to do it for URLs that are
absolutely known to be shortened URLs.  You have to keep a list of
known URL-shorteners.  It's hard/impossible to get a complete list,
but pretty easy to get a list of the big ones that cover 90% of all
shortened URLs you're likely to see.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 11:48 AM, Anthony Cartmell wrote:
>> Meanwhile - adding URI lookups (for URIs in the body of the domains)
>> and/or the option to add 3rd party URI list lookups - is STILL is
>> missing from MANY widely used anti-spam systems.
> If you mean following URLs in messages, you do need to be aware that
> this can break one-time login links.
>
> I have come across one local authority in the UK whose spam filter
> requests URLs in incoming messages. This meant that the spam filter had
> already used the one-time login link so that when the intended user
> tried they got an error saying that the link had already been used. Took
> a while to work out what was going wrong!


Sorry I wasn't more clear. I was just referring to standard things like 
usage of SURBL, URIBL, ivmURI, and Spamhaus's DBL list - where the 
domains and IPs that are in the body of the message are collected and 
checked against those lists - I didn't mean to imply that such links are 
actually followed. Again, sorry I wasn't more clear. I was just pointing 
out that if so many systems STILL don't do THAT - or provide an 
opportunity for adding additional such lists - then I doubt that we're 
going to see widespread adoption by mail systems of a process where, in 
real time spam filtering, they check to see where URL shortners lead to, 
and then factor that destination into the spam filtering.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Anthony Cartmell <aj...@fonant.com>]

> Meanwhile - adding URI lookups (for URIs in the body of the domains)
> and/or the option to add 3rd party URI list lookups - is STILL is
> missing from MANY widely used anti-spam systems.

If you mean following URLs in messages, you do need to be aware that
this can break one-time login links.

I have come across one local authority in the UK whose spam filter
requests URLs in incoming messages. This meant that the spam filter had
already used the one-time login link so that when the intended user
tried they got an error saying that the link had already been used. Took
a while to work out what was going wrong!

Anthony
-- 
www.fonant.com - Quality web sites
Tel. 01903 867 810
Fonant Ltd is registered in England and Wales, company No. 7006596
Registered office: Amelia House, Crescent Road, Worthing, West Sussex,
BN11 1QR

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 11:12 AM, Dianne Skoll wrote:
> Really?  This isn't rocket science.  If I thought of it, I'm sure dozens
> if not hundreds of others have thought of it and implemented it.

Meanwhile - adding URI lookups (for URIs in the body of the domains) 
and/or the option to add 3rd party URI list lookups - is STILL is 
missing from MANY widely used anti-spam systems. That isn't rocket 
science either - and is MUCH more needed and beneficial than this 
feature. That some others have made adjustments to their system to 
follow and act on redirectors during live spam filtering - again - 
doesn't alter my original point. The vast majority of anti-spam systems 
in the real world (1) don't (2) and won't any time soon. That is what I 
claimed. Please stop nitpicking and please stop arguing with a "straw man".

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 11:00:48 -0500
Rob McEwen <ro...@invaluement.com> wrote:

> > [Expanding shorteners] been part of our practice for about a year now.

> Excellent! I wish others would be as innovative and on top of things
> as you are! Unfortunately, your statement doesn't alter my point you
> were replying to, even one tiny bit.

Really?  This isn't rocket science.  If I thought of it, I'm sure dozens
if not hundreds of others have thought of it and implemented it.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 10:39 AM, Dianne Skoll wrote:
> We use HEAD requests to expand known URL-shorteners on a cluster that
> peaks around 60 msgs/s

Thanks for that information. That is good to know!

>> (b) and this isn't going to suddenly become a feature inside of many
>> types of spam filtering hardware and software overnight... that could
>> even take years,
> It's been part of our practice for about a year now.

Excellent! I wish others would be as innovative and on top of things as 
you are! Unfortunately, your statement doesn't alter my point you were 
replying to, even one tiny bit.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Wed, 21 Feb 2018 02:30:40 -0500
Rob McEwen <ro...@invaluement.com> wrote:

> (a) it might not "scale" for high volume mail flows and DNSBLs who,
> like invaluement, process dozens (or more) spams per second.

We use HEAD requests to expand known URL-shorteners on a cluster that
peaks around 60 msgs/s

> (b) and this isn't going to suddenly become a feature inside of many 
> types of spam filtering hardware and software overnight... that could 
> even take years,

It's been part of our practice for about a year now.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/21/2018 1:38 AM, @lbutlr wrote:
> As I suspected, it is possible to get the goo.gl target URL without loading the site, though using curl is probably not realistic in this specific case.


That is an idea worth exploring! Some might greatly benefit from that.

However:

(a) it might not "scale" for high volume mail flows and DNSBLs who, like 
invaluement, process dozens (or more) spams per second.

(b) and this isn't going to suddenly become a feature inside of many 
types of spam filtering hardware and software overnight... that could 
even take years, if it could ever even gain traction? It is taking into 
the decades just to get some of those software and hardware vendors to 
add support for URI blacklists, or support to for adding custom 3rd 
party URI blacklists. If that is taking literally decades - they are not 
going to add this feature within the foreseeable future.

So please don't think for a second that this somehow makes the plans I 
had described as unnecessary.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-20 (22:10 MST), Reindl Harald <h....@thelounge.net> wrote:
> 
> you may hit confirmation-urls (both ham and spam), trigger actions, trigger *one-time* urls which are invalid for the user after a dumb bot used them not talking about that it would be illegal in many countries in case of private ham-mails

As I suspected, it is possible to get the goo.gl target URL without loading the site, though using curl is probably not realistic in this specific case.

$ curl -s "http://goo.gl/ylUAd" | grep -o "http[^\"]*"
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

$ curl -s "http://bit.ly/savecastle" | grep -o "http[^\"]*"
http://community.livejournal.com/castle_tv/28872.html

Doesn't work with t.co, but that is not surprising since twitter uses that specifically to hide URLs, considering them all their property that must go through their servers.

-- 
Mos Eisley spaceport. You will never find a more wretched hive of scum
and villainy. We must be cautious.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-20 (08:30 MST), Rob McEwen <ro...@invaluement.com> wrote:
> 
> Spammers are starting to use this to evade spam filters,

This is not news. Spammers have been using shortness since 3 seconds after tinyurl.com launched.

> Keep in mind that, if a marketer is doing things the right way, they should have no need to obfuscate their own domain name. They should instead proudly use it and not feel the need to hide behind Google's shortner.

No, that is not at all true. The primary use of a shorter is to shorten a long URL to something that someone can type in.

Clicking a URL in an email is the height of stupidity, so having a short URL that someone can realistically type into a browser is much better.

> Yes, there are many legitimate uses of Google's shortner, too. However, we are now at a point where a VERY large % (a majority?) of uses of these headed to a typical user's mailbox are egregious spams, and a significant additional portion are likely-spams.


Any evidence of this?

> THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner, and we're amplifying other "bad" points.


Well, at least you are warning people. However, what you are doing is, frankly, dumb; if you think there's a huge problem, you can simply check the target URLs.

-- 
I've always had a flair for stage directions.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

>
> > >
> > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > require contribution. Please contact us informing your IP or range, for
> > > further details."
> >
> > This means, for example, your system do 10 queries at same second, then
> the
> > query frequency is 100ms.
>
> Yes, I got that bit.
>
> How big is an IP block?
>


Maybe I did not understand your question, but our DNSBL lists individual
IPs, even /128 for IPv6.

Sometimes our system lists entire blocks, like /64 for IPv6 and /24 for
IPv4.



>
> > > Please could you explain what this means; what limitations are imposed
> on
> > > use of this service - specifically what is an "IP block", and do you
> really
> > > mean "lower frequencies require contribution"?  Surely that should be
> > > "higher"?
> >
> > Yes, I am sure. Lets use the same example above, but now your system do
> 20
> > queries at same second, then the query frequency becomes 50ms, less than
> > first case.
>
> Ah; I would call 50ms the interval and 20 queries per second the frequency.
>
> Thanks for the explanation.
>
>
You are always welcome!

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Antony Stone <An...@spamassassin.open.source.it>]

On Tuesday 03 April 2018 at 16:09:38, Leandro wrote:

> 2018-04-03 10:34 GMT-03:00 Antony Stone:
> > On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:
> > > Hey guys. We just created an URL signature algorithm to be able to
> > > query an entire URL at our URIBL:
> > > 
> > > https://spfbl.net/en/uribl/
> > 
> > I don't think I understand the following statement on that page:
> > 
> > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > require contribution. Please contact us informing your IP or range, for
> > further details."
> 
> This means, for example, your system do 10 queries at same second, then the
> query frequency is 100ms.

Yes, I got that bit.

How big is an IP block?

> > Please could you explain what this means; what limitations are imposed on
> > use of this service - specifically what is an "IP block", and do you really
> > mean "lower frequencies require contribution"?  Surely that should be
> > "higher"?
> 
> Yes, I am sure. Lets use the same example above, but now your system do 20
> queries at same second, then the query frequency becomes 50ms, less than
> first case.

Ah; I would call 50ms the interval and 20 queries per second the frequency.

Thanks for the explanation.


Antony.

-- 
90% of networking problems are routing problems.
9 of the remaining 10% are routing problems in the other direction.
The remaining 1% might be something else, but check the routing anyway.

                                                   Please reply to the list;
                                                         please *don't* CC me.

Re: OT: Frequency vs. Period (was Re: The "goo.gl" shortner...)

[Leandro <le...@spfbl.net>]

2018-04-03 11:57 GMT-03:00 Dianne Skoll <df...@roaringpenguin.com>:

> On Tue, 3 Apr 2018 11:09:38 -0300
> Leandro <le...@spfbl.net> wrote:
>
> > This means, for example, your system do 10 queries at same second,
> > then the query frequency is 100ms.
>
> In SI units, frequency has the unit s^(-1) and period has the unit s,
> where s stands for "second"
>
> So 100ms is the period, and 10/s is the frequency.  Basic dimensional
> analysis.
>

You are right! My mistake. I just fixed website information. Thanks!


>
> Regards,
>
> Dianne.
>

Re: OT: Frequency vs. Period (was Re: The "goo.gl" shortner...)

["Kevin A. McGrail" <km...@apache.org>]

Note: Goo.gl is being shutdown.
https://www.engadget.com/2018/03/30/google-shutting-down-goo-gl-url-shortening-service/
Apologies if I already noted that here.

--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Tue, Apr 3, 2018 at 10:57 AM, Dianne Skoll <df...@roaringpenguin.com>
wrote:

> On Tue, 3 Apr 2018 11:09:38 -0300
> Leandro <le...@spfbl.net> wrote:
>
> > This means, for example, your system do 10 queries at same second,
> > then the query frequency is 100ms.
>
> In SI units, frequency has the unit s^(-1) and period has the unit s,
> where s stands for "second"
>
> So 100ms is the period, and 10/s is the frequency.  Basic dimensional
> analysis.
>
> Regards,
>
> Dianne.
>

OT: Frequency vs. Period (was Re: The "goo.gl" shortner...)

[Dianne Skoll <df...@roaringpenguin.com>]

On Tue, 3 Apr 2018 11:09:38 -0300
Leandro <le...@spfbl.net> wrote:

> This means, for example, your system do 10 queries at same second,
> then the query frequency is 100ms.

In SI units, frequency has the unit s^(-1) and period has the unit s,
where s stands for "second"

So 100ms is the period, and 10/s is the frequency.  Basic dimensional
analysis.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

>
> > > Then the frequency is 10 per second, not 100ms. Querying more often
> > > is a higher frequency.
> >
> > That is it! 10 per second or one every 100ms. The first is a flow rate
> and
> > the second is a frequency.
>
> One every 100ms is a frequency, agreed.
>
> Two every 100ms is a higher frequency, and means faster requests.
>
> One every 50ms is the same rate as two every 100ms, therefore it is also a
> higher frequency than one every 100ms.
>

You are right! My mistake. I just fixed website information. Thanks!


>
> Regards,
>
>
> Antony.
>
> --
> I wasn't sure about having a beard at first, but then it grew on me.
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Antony Stone <An...@spamassassin.open.source.it>]

On Tuesday 03 April 2018 at 16:43:09, Leandro wrote:

> 2018-04-03 11:35 GMT-03:00 RW:
> > On Tue, 3 Apr 2018 11:09:38 -0300 Leandro wrote:
> > > 2018-04-03 10:34 GMT-03:00 Antony Stone:
> > > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > > require contribution. Please contact us informing your IP or range,
> > > > for further details."
> > > 
> > > This means, for example, your system do 10 queries at same second,
> > > then the query frequency is 100ms.
> > 
> > Then the frequency is 10 per second, not 100ms. Querying more often
> > is a higher frequency.
> 
> That is it! 10 per second or one every 100ms. The first is a flow rate and
> the second is a frequency.

One every 100ms is a frequency, agreed.

Two every 100ms is a higher frequency, and means faster requests.

One every 50ms is the same rate as two every 100ms, therefore it is also a 
higher frequency than one every 100ms.

Regards,


Antony.

-- 
I wasn't sure about having a beard at first, but then it grew on me.

                                                   Please reply to the list;
                                                         please *don't* CC me.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-04-03 11:35 GMT-03:00 RW <rw...@googlemail.com>:

> On Tue, 3 Apr 2018 11:09:38 -0300
> Leandro wrote:
>
> > 2018-04-03 10:34 GMT-03:00 Antony Stone <
> > Antony.Stone@spamassassin.open.source.it>:
>
> > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > require contribution. Please contact us informing your IP or range,
> > > for further details."
> > >
> >
> >
> > This means, for example, your system do 10 queries at same second,
> > then the query frequency is 100ms.
>
> Then the frequency is 10 per second, not 100ms. Querying more often
> is a higher frequency.
>


That is it! 10 per second or one every 100ms. The first is a flow rate and
the second is a frequency.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[RW <rw...@googlemail.com>]

On Tue, 3 Apr 2018 11:09:38 -0300
Leandro wrote:

> 2018-04-03 10:34 GMT-03:00 Antony Stone <
> Antony.Stone@spamassassin.open.source.it>:  

> > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > require contribution. Please contact us informing your IP or range,
> > for further details."
> >  
> 
> 
> This means, for example, your system do 10 queries at same second,
> then the query frequency is 100ms.

Then the frequency is 10 per second, not 100ms. Querying more often
is a higher frequency.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-04-03 10:34 GMT-03:00 Antony Stone <
Antony.Stone@spamassassin.open.source.it>:

> On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:
>
> > Hey guys. We just created an URL signature algorithm to be able to query
> an
> > entire URL at our URIBL:
> >
> > https://spfbl.net/en/uribl/
>
> I don't think I understand the following statement on that page:
>
> "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies require
> contribution. Please contact us informing your IP or range, for further
> details."
>


This means, for example, your system do 10 queries at same second, then the
query frequency is 100ms.



>
> Please could you explain what this means; what limitations are imposed on
> use
> of this service - specifically what is an "IP block", and do you really
> mean
> "lower frequencies require contribution"?  Surely that should be "higher"?
>


Yes, I am sure. Lets use the same example above, but now your system do 20
queries at same second, then the query frequency becomes 50ms, less than
first case.



>
>
> Thanks,
>
>
> Antony.
>
> --
> There's a good theatrical performance about puns on in the West End.  It's
> a
> play on words.
>
>                                                    Please reply to the
> list;
>                                                          please *don't* CC
> me.
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Antony Stone <An...@spamassassin.open.source.it>]

On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:

> Hey guys. We just created an URL signature algorithm to be able to query an
> entire URL at our URIBL:
> 
> https://spfbl.net/en/uribl/

I don't think I understand the following statement on that page:

"IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies require 
contribution. Please contact us informing your IP or range, for further 
details."

Please could you explain what this means; what limitations are imposed on use 
of this service - specifically what is an "IP block", and do you really mean 
"lower frequencies require contribution"?  Surely that should be "higher"?


Thanks,


Antony.

-- 
There's a good theatrical performance about puns on in the West End.  It's a 
play on words.

                                                   Please reply to the list;
                                                         please *don't* CC me.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

>
> We just created an URL signature algorithm to be able to query an entire
> URL at our URIBL:
>
> https://spfbl.net/en/uribl/
>
> Now we are able to blacklist any malicious shortener URL
>
>
> Leandro,
>
> Thanks for all you do! And good luck with that. But there are a few
> potential problems. When I analyzed Google's shortners about a month ago, I
> found that a VERY large percentage of the most malicious shortened URLs
> were a situation where the spammers were generating a unique shortner for
> each individual message/recipient-address. This causes the following HUGE
> problems (at least for THESE particular shortners) when publishing a
> full-URL dnsbl:
>

Thank you for all those observations!


> (1) much of what you populate your rbldnsd file with is going to be
> totally ineffective for anyone since it ONLY applied to whatever single
> email address where the spam was original sent (where you had trapped it) -
> everyone else is going to get DIFFERENT shortners for the spam from these
> same campaigns that are sent to their users.
>

You are right, but we do not use rbldnsd. We have our own DNSBL
implementation that uses a more efficient data structure. Anyway, I thing
that is not a good idea list each shortener, as you sad. Maybe thrice
complains of same shortener. We will discover what is the best some time.


> (2) get ready for EXTREME rbldnsd bloat. You're gonna need a LOT of RAM
> eventually? And if you ever distribute via rsync, those are going to be
> HUGE rsync files (and then THEY will need a lot of RAM). Sadly, most of
> that bloat is going to come from entries that are doing absolutely nothing
> for anyone.
>

That is it! We use a VM with 16GB and the software is using about 10GB to
keep more than 30 million registers at memory. That is something about 350
bytes per register. Our software have an expiration mechanism, than this
memory occupation is not growing to fast now. But we must keep one eye on
it always.


> (3) You might be revealing your spam traps to the spammers. In cases where
> the spammers are sending that 1-to-1 spam to single recipient shortners,
> then all they gave to do is enumerate through their list of shortners,
> checking them against your list - and they INSTANTLY get a list of every
> recipient address that triggers a listing on your DNSBL. If you want to
> destroy the effectiveness of your own DNSBL's spam traps - be my guest. But
> if you're getting 3rd party spam feeds (paid or free) - then know that
> you're then screwing over your 3rd party spam feed's spam traps - and those
> OTHER anti-spam system that rely on such feeds, which will then diminish in
> quality. (unless you are filtering OUT these MANY 1-to-1 shortner spams)
>

Not only spamtraps will trigger this listing. All active users will do it
too by complains. The spammer will not know who is spamtrap and who is
active user.


> Maybe there is enough OTHER shortners (that are sending the same shortners
> to multiple recipients) to make this worthwhile? But the bloat from the
> ones that are uniquely generated could be a challenge, and could
> potentially cause a MASSIVE amount of useless queries. I'd be very
> interested to see what PERCENTAGE of such queries generated a hit!
>
> Meanwhile, in my analysis I did about a month ago, about 80% of Google's
> shortners found in egregious spams (that did this one-to-one
> shorter-to-recipient tactic)... were all banging on one of ONLY a dozen
> different spammers' domains. Therefore, doing a lookup on these and then
> checking the domain found at the base of the link it redirects to... is a
> more effective strategy for these - whereas, for THESE 80% of egregious
> google shortners, a full URL lookup is worthless, consuming resources
> without a single hit.
>

That is right. We have same situation here.

But check first URL is not only action we do. Our script can follow
shortener redirections and catch the spammer by last URL of redirection
chain:

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0

The spammers can be trapped by any shortener they have or by this dozen
domains that shortener hides.

Alternatively, you may have found a way to filter out these types of
> individualized shortners, to prevent that bloat? But even then, everyone
> should know that while your new list might be helpful, it would be good for
> others to know your new list isn't applicable to a large percentage of
> spammy shortners, since it is still useless against these individualized
> shortners.
>

I think that we all must cause to much of work for spammers, as much they
cause to us. If the spammer uses individualized shorteners, we can list
each one by crossing data with listed final chain URL domains. If they uses
individualized URL domains, we can list each one by crossing data with
listed URL equivalent IP (same machine for all spammer domains). We can
make it more and more expensive for spammers. But we must work together to
do it.


> NOTE: Google has made some improvements recently, and I haven't yet
> analyzed how much those improvements have changed any of these things I've
> mentioned?
>
> PS - the alphanumeric code at the end of these shortners tend to be
> case-sensitive, while the rest of the URL is NOT case sensitive (and they
> also work with both "https" and "http")... so you might want to standardize
> this on (1) https and (2) everything lower case up until the code at the
> end of the shortner - before the MD5 is calculated. Otherwise, it could
> easily break if the spammer just mixes up the capitalization of the
> shortner URL up until the code at the end of the shortner.
>

Great! I did not thought this. This is a new problem and your solution is
very good. Let's discuss more about this idea before any implementation.


> --
> Rob McEwenhttps://www.invaluement.com
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Dianne Skoll <df...@roaringpenguin.com>]

On Tue, 3 Apr 2018 11:21:35 -0400
Rob McEwen <ro...@invaluement.com> wrote:

> Thanks for all you do! And good luck with that. But there are a few 
> potential problems. When I analyzed Google's shortners about a month 
> ago, I found that a VERY large percentage of the most malicious 
> shortened URLs were a situation where the spammers were generating a 
> unique shortner for each individual message/recipient-address.

We found that too, but in most cases, they generated the unique URLs
by adding query parameters to the same base URL, sort of like this:

http://malware.net/?id=znsjdsjau
http://malware.net/?id=aosu94e
etc...

and then shortening them.

So if you blacklist just the base URL, you cover those all off,
assuming you expand out shortened URLs as part of your processing, of course.

> Meanwhile, in my analysis I did about a month ago, about 80% of
> Google's shortners found in egregious spams (that did this one-to-one 
> shorter-to-recipient tactic)... were all banging on one of ONLY a
> dozen different spammers' domains. Therefore, doing a lookup on these
> and then checking the domain found at the base of the link it
> redirects to... is a more effective strategy for these - whereas, for
> THESE 80% of egregious google shortners, a full URL lookup is
> worthless, consuming resources without a single hit.

Yep, that's what we found too.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 4/3/2018 9:27 AM, Leandro wrote:
> We just created an URL signature algorithm to be able to query an 
> entire URL at our URIBL:
>
> https://spfbl.net/en/uribl/
>
> Now we are able to blacklist any malicious shortener URL


Leandro,

Thanks for all you do! And good luck with that. But there are a few 
potential problems. When I analyzed Google's shortners about a month 
ago, I found that a VERY large percentage of the most malicious 
shortened URLs were a situation where the spammers were generating a 
unique shortner for each individual message/recipient-address. This 
causes the following HUGE problems (at least for THESE particular 
shortners) when publishing a full-URL dnsbl:

(1) much of what you populate your rbldnsd file with is going to be 
totally ineffective for anyone since it ONLY applied to whatever single 
email address where the spam was original sent (where you had trapped 
it) - everyone else is going to get DIFFERENT shortners for the spam 
from these same campaigns that are sent to their users.

(2) get ready for EXTREME rbldnsd bloat. You're gonna need a LOT of RAM 
eventually? And if you ever distribute via rsync, those are going to be 
HUGE rsync files (and then THEY will need a lot of RAM). Sadly, most of 
that bloat is going to come from entries that are doing absolutely 
nothing for anyone.

(3) You might be revealing your spam traps to the spammers. In cases 
where the spammers are sending that 1-to-1 spam to single recipient 
shortners, then all they gave to do is enumerate through their list of 
shortners, checking them against your list - and they INSTANTLY get a 
list of every recipient address that triggers a listing on your DNSBL. 
If you want to destroy the effectiveness of your own DNSBL's spam traps 
- be my guest. But if you're getting 3rd party spam feeds (paid or free) 
- then know that you're then screwing over your 3rd party spam feed's 
spam traps - and those OTHER anti-spam system that rely on such feeds, 
which will then diminish in quality. (unless you are filtering OUT these 
MANY 1-to-1 shortner spams)

Maybe there is enough OTHER shortners (that are sending the same 
shortners to multiple recipients) to make this worthwhile? But the bloat 
from the ones that are uniquely generated could be a challenge, and 
could potentially cause a MASSIVE amount of useless queries. I'd be very 
interested to see what PERCENTAGE of such queries generated a hit!

Meanwhile, in my analysis I did about a month ago, about 80% of Google's 
shortners found in egregious spams (that did this one-to-one 
shorter-to-recipient tactic)... were all banging on one of ONLY a dozen 
different spammers' domains. Therefore, doing a lookup on these and then 
checking the domain found at the base of the link it redirects to... is 
a more effective strategy for these - whereas, for THESE 80% of 
egregious google shortners, a full URL lookup is worthless, consuming 
resources without a single hit.

Alternatively, you may have found a way to filter out these types of 
individualized shortners, to prevent that bloat? But even then, everyone 
should know that while your new list might be helpful, it would be good 
for others to know your new list isn't applicable to a large percentage 
of spammy shortners, since it is still useless against these 
individualized shortners.

NOTE: Google has made some improvements recently, and I haven't yet 
analyzed how much those improvements have changed any of these things 
I've mentioned?

PS - the alphanumeric code at the end of these shortners tend to be 
case-sensitive, while the rest of the URL is NOT case sensitive (and 
they also work with both "https" and "http")... so you might want to 
standardize this on (1) https and (2) everything lower case up until the 
code at the end of the shortner - before the MD5 is calculated. 
Otherwise, it could easily break if the spammer just mixes up the 
capitalization of the shortner URL up until the code at the end of the 
shortner.

-- 
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

2018-04-03 10:27 GMT-03:00 Leandro <le...@spfbl.net>:

> Hey guys. We just created an URL signature algorithm to be able to query
> an entire URL at our URIBL:
>
> https://spfbl.net/en/uribl/
>
> Now we are able to blacklist any malicious shortener URL. Now I will think
> about some public complain interface that automatic lists any correct
> malicious sample using some simple AI.
>


Hi. I just implemented automatic URL listing for these criteria:

   1. Shortener URL that redirect to other shortener (evasion) and
   2. Shortener URL that redirect to a bad reputation URL domain (hiding).

The shortener URL will be automatically listed if queried at this tool:

https://matrix.spfbl.net/en/

I ask that you guys test this new feature and call me if any bug was found.

I will think at another objective criteria to add at this interface, for
automatic URL listings. Maybe you can help me think at one.

The goal of this feature is never depend on a human to check these URLs
listings, if possible, and be an alternative for services like Phishtank
and others.

To have a very fast abusive URL listings and quick public exposure of
malicious URLs.



>
> All you have to do now is implement a SA plugin to make this signature and
> do the URIBL query.
>


If anyone can do it. I will be very grateful!



>
> Regards,
>
> Leandro
> SPFBL.net
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Leandro <le...@spfbl.net>]

Hey guys. We just created an URL signature algorithm to be able to query an
entire URL at our URIBL:

https://spfbl.net/en/uribl/

Now we are able to blacklist any malicious shortener URL. Now I will think
about some public complain interface that automatic lists any correct
malicious sample using some simple AI.

All you have to do now is implement a SA plugin to make this signature and
do the URIBL query.

Regards,

Leandro
SPFBL.net

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 4/1/2018 7:10 PM, Kevin A. McGrail wrote:
> No, I don't think it's an April Fool's trick though it is possible. 
> They announced this a day or 2 ago.
> See 
> https://www.cloudconnectcommunity.com/ccc/ls/community/g-suite-feature-ideas/post/6320666165116928 
> and https://firebase.google.com/docs/dynamic-links/


I have been in talks with Google about this, sharing with them specific 
data and stats. My contact there sent me an email about this a day or 
two ago referencing that announcement - I just hadn't had time to review 
it yet. So this is for real.

The data I had compiled and shared with them that I gathered about a 
month ago... was devastating. I can't make that report public because 
the spammers were OFTEN sending a one-to-one ratio of a single 
individual google shortener per recipient address. Therefore, making 
that data public would have revealed many spamtrap addresses (both mine 
and those of 3rd party spam feeds). Another problem with that data... is 
that I was finding 10s of thousands of google shortners that were 
banging on the same dozens spammer's domains - and it would be so 
trivial for Google to look for that pattern and then just wipe out all 
of THOSE many shortners that were redirecting to one of these dozen 
spammers' domains. At the same time, the vast majority of these 
shortners were persisting for weeks and weeks with no sign that Google 
was hardly lifting a finger to terminate them. In fact, in that initial 
report, 95+% of shortners used by spammers were persisting for weeks on 
end, and 80% of those were using one of those 12 spammer' domains.

HOWEVER... I started compiling new data few days ago, and it looks MUCH 
better now. I'm not done putting that report together, but early 
indications show that something has recently changed for the better - 
but that there is still significant room for improvement, but they are 
at least headed in the right direction. But that report is still under 
construction.

So I think the pressure on Google, along with some of the data I 
provided them... might have helped? Or maybe that was just "one straw 
that broke the camel's back"? Either way, I'm happy that this seems to 
be getting fixed, or they are at least headed in the right direction.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["Kevin A. McGrail" <km...@apache.org>]

No, I don't think it's an April Fool's trick though it is possible.  They
announced this a day or 2 ago.

See
https://www.cloudconnectcommunity.com/ccc/ls/community/g-suite-feature-ideas/post/6320666165116928
and https://firebase.google.com/docs/dynamic-links/

Regards,
KAM

--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Sun, Apr 1, 2018 at 5:02 PM, Benny Pedersen <me...@junc.eu> wrote:

> @lbutlr skrev den 2018-04-01 22:46:
>
>> On 2018-02-20 (08:30 MST), Rob McEwen <ro...@invaluement.com> wrote:
>>
>>>
>>> RE: The "goo.gl
>>> " shortner is OUT OF CONTROL (+ invaluement's response)
>>>
>>
>> <https://tidbits.com/2018/04/01/google-sunsets-goo-gl-url-shortener/>
>>
>
> april fools day :=)
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Benny Pedersen <me...@junc.eu>]

@lbutlr skrev den 2018-04-01 22:46:
> On 2018-02-20 (08:30 MST), Rob McEwen <ro...@invaluement.com> wrote:
>> 
>> RE: The "goo.gl
>> " shortner is OUT OF CONTROL (+ invaluement's response)
> 
> <https://tidbits.com/2018/04/01/google-sunsets-goo-gl-url-shortener/>

april fools day :=)

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

["@lbutlr" <kr...@kreme.com>]

On 2018-02-20 (08:30 MST), Rob McEwen <ro...@invaluement.com> wrote:
> 
> RE: The "goo.gl
> " shortner is OUT OF CONTROL (+ invaluement's response)

<https://tidbits.com/2018/04/01/google-sunsets-goo-gl-url-shortener/>


-- 
Technically, Aziraphale was a Principality, but people made jokes about
that these days

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/20/2018 11:45 AM, Rob McEwen wrote:
> And we ALL have to constantly shift our tactics to deal with emerging 
> realities like this one - or risk getting left behind by our 
> competitors who do keep up.


ALSO - Likewise, it was very frustrating that I had to spend hours late 
last night making adjustments to my spam filter to be able to block more 
of these egregious spams that are often so difficult for a spam filter 
to block - particularly since extra careful measures are needed to keep 
such adjustments from blocking legitimate messages. (I've been trying to 
get to that for weeks, as this problem has been festering for some time 
- but I just recently recovered from the flu.)

So nobody should cry on my shoulder about the difficulty of ESPs doing 
such abuse monitoring/prevention. Any such person doing so is likely 
underestimating the current size and frustration that spam filtering 
admins are having with this problem!

-- 
Rob McEwen
https://www.invaluement.com
  

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/20/2018 12:21 PM, Reindl Harald wrote:
> we have well working outbound spamfilters

Excellent!

> but just because someone has a google-shortener within a mail says 
> *nothing at all* - frankly i even got a week ago a mail from my boss 
> where the google-shortener was used for a only internal reachable 
> server with a long list of params in the url
> and hence the google-shortener don't say anything 


This is a very abused loophole that says MUCH in certain contexts. And 
I've carefully constructed these change at invaluement to be extremely 
unlikely to impact those who are using "goo .gl" for legitimate purposes 
and are not using it to cloak their domain in messages that are UBE or 
otherwise not desired by the recipients. But, in contrast, marketers 
and/or ESPs who start doing this routingly, as a purposeful regular 
practice, and who don't have some kind of real and specific purpose such 
as what that you described, are essentially giving DNSBLs and spam 
filters "the middle finger". So I'm giving it back. ANYTHING that 
facilitates anonymizing identity is VERY BAD for email. Facilitating 
anonymizing identity causes more spam to be delivered and punishes good 
senders when bad senders get away with that. Methods that facilitates 
anonymizing identity for email is not something that anyone should 
defend or celebrate - even if anonymizing identity wasn't the original 
intended purpose. I understand your very legitimate concern that this 
crackdown might lead to collateral damage. That is admirable. But 
acceptance of a new and pervasive situation in email that anonymizes 
identity is a HUGE step backwards... like going back to the mid 2000s, 
or something. So some "push back" measures are exceedingly warranted.

-- 
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

[Rob McEwen <ro...@invaluement.com>]

On 2/20/2018 10:57 AM, Reindl Harald wrote:
> and how do you imagine that i prevent paying customers to use whatever 
> url-shortener?

Perhaps use the SAME methods that an ESP would use to prevent a customer 
from sending an egregious phish (or terminate their account for sending 
a phish). Of course, I also recognize that an egregious phish is much 
worse. But my point is that such abuse monitoring and prevention is a 
real thing for ESPs! Yes, some ESPs are more sophisticated than others, 
where they do a better job at this than others. For example, I've 
received two egregious phishes to my own email address, from MailGun IP 
space, within the past several months. I alerted them in both instances 
and hopefully they are improving their system? In contrast, I don't 
think I've ever seen such a phish from Exact Target, from example. That 
isn't by accident! Some do a better job of this than others. And even 
though no ESP can be perfect - that doesn't mean they can't improve. And 
we ALL have to constantly shift our tactics to deal with emerging 
realities like this one - or risk getting left behind by our competitors 
who do keep up.

Also, getting ESPs to pass this message on to their clients, even if 
just adding this to their instructions for clients, even if just as a 
"best practices" warning... might also go a long way.

> when you start list to many legit servers because of that you RBL will 
> be no longer useable for responsible admins which primary job is 
> receiove and deliver email and not to reject it 


I'm extremely confident that this won't happen. Most likely, a few 
marginal ESPs and marketers will get blacklisted who were previously 
just barely avoiding detection. Also, we OFTEN get outliers (such as an 
occasional VERY bad spam that came from a normally VERY good sender), 
and "decoys", too! In those cases, if those messages had led to an 
automatic blacklisting, and we didn't FIRST check those domains and IPs 
against our very sophisticated "false positive prevention filter" - then 
what you described - would have happened a long time ago already. But, 
instead, invaluement's reputation for low False Positives speaks for 
itself. Given what I know about how invaluement works "under the hood", 
I can say with confidence that it is practically impossible for this 
change to put a dent in our hard-earned low-FP reputation. But this 
COULD cause problems for some already dark-gray-hat ESPs who let this 
practice run rampant.

-- 
Rob McEwen
https://www.invaluement.com