You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Dianne Skoll <df...@roaringpenguin.com> on 2018/02/21 15:37:29 UTC
Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
On Tue, 20 Feb 2018 23:38:53 -0700
"@lbutlr" <kr...@kreme.com> wrote:
> As I suspected, it is possible to get the goo.gl target URL without
> loading the site, though using curl is probably not realistic in this
> specific case.
We do a HEAD request and it works on most URL shorteners.
The concern voiced in another email about overloading Google's
infrastructure is quite charming and quaint.
Regards,
Dianne.
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
Posted by Benny Pedersen <me...@junc.eu>.
Dianne Skoll skrev den 2018-02-21 16:37:
> We do a HEAD request and it works on most URL shorteners.
>
> The concern voiced in another email about overloading Google's
> infrastructure is quite charming and quaint.
+1
some with icla could add this to spamasssassin with
https://github.com/smfreegard/DecodeShortURLs
so if yes problem is solved imho, but should we consider all _URIHOSTS_
as shortners ?
unless thay are confirmed not to do this
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
Posted by Rob McEwen <ro...@invaluement.com>.
On 2/21/2018 11:44 AM, Dianne Skoll wrote:
> On Wed, 21 Feb 2018 16:35:27 +0000
> Karol Augustin <ka...@augustin.pl> wrote:
>> I think the point here might be that if Google acted promptly on abuse
>> spammers would stop using shorteners.
> True, that might happen. OTOH, I see about as many spams with bit.ly
> shorteners as goo.gl shorteners which is not what one might expect if
> bit.ly were really that much more proactive than goo.gl.
I'm sure mileage may vary - but I'm seeing about 10X the abuse for
goo.gl right now as I see from bit.ly. Also, when I do random checks on
a handful of abused bitly shortners, a high percentage of them are
already terminated. But when I do random checks of abused goo.gl
redirectors, most of them are still operational. (I'm referring to
redirectors found in spams within the previous few days of when I
checked them, with at least hours having gone by since the message was
sent - I know that sounds anecdotal - but as I've been researching this
in the past weeks, that pattern keeps happening). One thing that could
potentially make those numbers different - is when you compare one
system that blocks MUCH spam at the perimeter based on the sending IP,
such as blocking all Zen-listed spams before DATA.... while another
system might capture ALL messages and process them all. The latter is
what my system does. That also might explain the difference in stats?
--
Rob McEwen
https://www.invaluement.com
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT
OF CONTROL (+ invaluement's response))
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Wed, 21 Feb 2018 16:35:27 +0000
Karol Augustin <ka...@augustin.pl> wrote:
> I think the point here might be that if Google acted promptly on abuse
> spammers would stop using shorteners.
True, that might happen. OTOH, I see about as many spams with bit.ly
shorteners as goo.gl shorteners which is not what one might expect if
bit.ly were really that much more proactive than goo.gl.
Regards,
Dianne.
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
Posted by Karol Augustin <ka...@augustin.pl>.
On 2018-02-21 16:31, Dianne Skoll wrote:
> On Wed, 21 Feb 2018 11:29:00 -0500
> Rob McEwen <ro...@invaluement.com> wrote:
>
>> Nevertheless, it is a shame to have to shift more of the burden onto
>> spam filters to do more work (some of which requires MORE latency) -
>> in order to partly mitigate Google's failure to prevent/correct the
>> abuse.
>
> Yes, I agree. On the other hand, IMO a spam filter should block messages
> that point to a taken-down shortened URL. Although such messages may
> not be downright dangerous, they are still annoying and are still spam.
> There's no way of avoiding the work.
>
I think the point here might be that if Google acted promptly on abuse
spammers would stop using shorteners.
Karol
--
Karol Augustin
karol@augustin.pl
http://karolaugustin.pl/
+353 85 775 5312
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT
OF CONTROL (+ invaluement's response))
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Wed, 21 Feb 2018 11:29:00 -0500
Rob McEwen <ro...@invaluement.com> wrote:
> Nevertheless, it is a shame to have to shift more of the burden onto
> spam filters to do more work (some of which requires MORE latency) -
> in order to partly mitigate Google's failure to prevent/correct the
> abuse.
Yes, I agree. On the other hand, IMO a spam filter should block messages
that point to a taken-down shortened URL. Although such messages may
not be downright dangerous, they are still annoying and are still spam.
There's no way of avoiding the work.
Regards,
Dianne.
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
Posted by Rob McEwen <ro...@invaluement.com>.
On 2/21/2018 11:11 AM, Dianne Skoll wrote:
> I guess I misinterpreted: "...such automated lookups could also put a
> huge extra burden on Google's servers..." from Message-Id
> <b2...@invaluement.com>
Oh yeah, I'd forgotten about that part. it was a more minor point. But
as I think back on my thought processes at the time I typed those words
- I was envisioning what would happen if ALL ISPs and hosters and spam
filtering vendors, SA installations etc... ALL started doing all of
those lookups. But yeah, maybe that would still be a drop in the bucket
compared to all that Google does.
Nevertheless, it is a shame to have to shift more of the burden onto
spam filters to do more work (some of which requires MORE latency) - in
order to partly mitigate Google's failure to prevent/correct the abuse.
In contrast, at this time, bit.ly is running circles around Google in
terms of quickly shutting down their abused redirectors. I know this
isn't easy, but there is definitely room for improvement.
But my larger point in that overall post you quoted from, was my concern
about one organization doing high volume lookups from a single server
getting blocked or captcha'd.
--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT
OF CONTROL (+ invaluement's response))
Posted by Dianne Skoll <df...@roaringpenguin.com>.
On Wed, 21 Feb 2018 10:58:17 -0500
Rob McEwen <ro...@invaluement.com> wrote:
> On 2/21/2018 10:37 AM, Dianne Skoll wrote:
> > The concern voiced in another email about overloading Google's
> > infrastructure is quite charming and quaint.
> My concern was NEVER about overloading google.
I guess I misinterpreted: "...such automated lookups could also put a
huge extra burden on Google's servers..." from Message-Id
<b2...@invaluement.com>
> My concern was about Google auto-blocking or throwing a captcha at
> very high volume and automated lookups. That is a HUGE difference.
That's a concern, but you can mitigate it slightly by caching the
results of URL expansion. Also, I suspect most URL-shorteners
expect to be bombarded with requests and are engineered accordingly.
I have seen a CAPTCHA thrown up from the search page, but I've never
seen a goo.gl HEAD request reply with anything other than 404 or 301.
Regards,
Dianne.
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF
CONTROL (+ invaluement's response))
Posted by Rob McEwen <ro...@invaluement.com>.
On 2/21/2018 10:37 AM, Dianne Skoll wrote:
> The concern voiced in another email about overloading Google's
> infrastructure is quite charming and quaint.
My concern was NEVER about overloading google. My concern was about
Google auto-blocking or throwing a captcha at very high volume and
automated lookups. That is a HUGE difference.
--
Rob McEwen
https://www.invaluement.com