The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Dianne Skoll <df...@roaringpenguin.com>]

On Mon, 30 Jan 2017 09:06:34 -0500
Rob McEwen <ro...@invaluement.com> wrote:

> On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:
> > they do and it has been mentioned:
> > https://help.yahoo.com/kb/SLN23997.html

Cool.  So Yahoo uses an HTML page that's a pain to process by
computer.  Microsoft has https://support.content.office.net/en-us/static/O365IPAddresses.xml,
which at least is XML.  And Google, so far as I can see, can be mined by
recursively expanding _spf.google.com.

Yay standards...

Regards,

Dianne.

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[David Jones <dj...@ena.com>]

>From: Dianne Skoll <df...@roaringpenguin.com>
    
>On Mon, 30 Jan 2017 09:06:34 -0500
>Rob McEwen <ro...@invaluement.com> wrote:

>> On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:
>> > they do and it has been mentioned:
>> > https://help.yahoo.com/kb/SLN23997.html

>Yahoo Outbound IP addresses | Yahoo Help - SLN23997
>help.yahoo.com
>Yahoo Outbound IP addresses. If you're looking for a list of IP addresses that Yahoo Mail sends emails from, we >have them for you below. Just click a link below to ...

Quick and dirty (I know there are many different ways to do this
so I am not saying this is the only way -- no flaming please.):

elinks -dump https://help.yahoo.com/kb/SLN23997.html | grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}(\/([0-9]|[1-2][0-9]|3[0-2]))?' | awk '{print $1}'

>Cool.  So Yahoo uses an HTML page that's a pain to process by
>computer.  Microsoft has  https://support.content.office.net/en-us/static/O365IPAddresses.xml,
>which at least is XML.  And Google, so far as I can see, can be mined by
>recursively expanding _spf.google.com.

Everyone else that I have needed to whitelist in postcreen with
postwhite will work fine by recursively expanding out their
TXT SPF record which is exactly what postwhite does.

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Rob McEwen <ro...@invaluement.com>]

On 2/1/2017 12:56 AM, Dave Warren wrote:
> They publish SPF records and DKIM sign everything for competent SMTP
> receivers to handle in real-time, AND they publish a HTML version for
> humans, and yet someone still finds a reason to complain?

Dave,

After the initial question was raised, it took about 11 posts and almost 
24 hours for someone to notice the discussion who happened to know about 
the "HTML version for humans" and mention that. During those 11 posts, a 
well-respected and knowledgeable person was actually defending Yahoo for 
NOT having such a page, which gave the impression that such didn't 
exist. (certainly, that was a head-fake that I fell for, even if such 
was very innocent)

So I think there is a strong argument that the existence of this page 
page isn't exactly common knowledge. Archive.org suggests that this page 
has only existed for a couple of years. I've been looking for it 
(occasionally) for the past 10 years - so I think all my memories of 
past discussions in past years about such a page not existing - were 
probably accurate. By the time this page existed, I had given up on 
finding it. (not that I spend every waking hour looking for it - I think 
I probably looked for it about once every year or two - for some time - 
and the need for this isn't so great with other senders - because few 
senders [even large ones] have such a MASSIVE amount of sending IPs that 
are so particularly hard to find)

Regarding your references about such a page not being needed - all I'm 
going to say is that some systems benefit from having large IP ranges 
preemptively whitelisted for the sake of efficiency. There are scenarios 
in certain very high volume systems where this enables the processing of 
messages at order of magnitudes faster rates than if SPF and DKIM and 
FCrDNS-confirmation had to be checked on every sending IP. MUCH of that 
relies on the response times of 3rd party servers - which (even at 
best!) is order of magnitudes slower than a local rbldnsd query  - or 
than an optimized binary search of an in-memory array - which is even 
faster than rbldnsd or even a high-end in-memory database. Sometimes, 
such 3rd party servers can "freeze up" in their responses, or rate limit 
queries - or firewall such lookups for what is perceived as abuse - 
causing further complications. Caching only does so much to prevent this!

That kind of need for speed is the world in which I live. At 
invaluement, I'm processing dozens of spams per second - and since much 
of these are ones where the "low-hanging fruit" - such as ALREADY 
heavily blacklisted botnet-sent spams are ALREADY filtered out before 
they get to my system - that means that the processing resources per 
spam is already much higher for my system than that of a typical ISP or 
hoster's natural incoming spam. (I process a higher concentration of the 
more sneaky spams and the newer emitters)

With this in mind... if I deleted my IP whitelist, and had to rely on 
SPF and DKIM and FCrDNS-verification for EVERY message, my queues would 
back up considerably - and a lot of worthy blacklistings of IPs and 
domains from new incoming spams would get considerably delayed. (again, 
inevitably - at this volume - issues come up where such 
queries/verification suddenly "freeze up" or get rate limited, 
firewalled, etc)

And I think my need for efficiency is probably not much different than 
some very large hosters and ISPs - who process mail for millions of users?

And I think we've already established that there is no possible way to 
generate "on demand" and remotely efficiently the information on that 
HTML page just via Yahoo's SPF records.

iow - maybe you should have a little more respect and try to be a little 
less snarky in the future - when you don't necessarily know/understand 
others' situation/requirements that may be a little different than your 
particular situation/requirements.

-- 
Rob McEwen

Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)

[Dave Warren <da...@hireahit.com>]

On 2017-01-30 08:06, Dianne Skoll wrote:
> On Mon, 30 Jan 2017 09:06:34 -0500
> Rob McEwen <ro...@invaluement.com> wrote:
>
>> On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote:
>>> they do and it has been mentioned:
>>> https://help.yahoo.com/kb/SLN23997.html
> Cool.  So Yahoo uses an HTML page that's a pain to process by
> computer.

They publish SPF records and DKIM sign everything for competent SMTP 
receivers to handle in real-time, AND they publish a HTML version for 
humans, and yet someone still finds a reason to complain?

Maybe it's just me, but hand-maintaining a list of IPs to whitelist is 
so 1997s. The real value of SPF and DKIM is that you don't do any of 
that, you can whitelist by domain and let the sending domain tell you, 
in real time, whether or not the inbound message should be trusted.

Or, if you insist on doing things manually, glance at the HTML source 
and spend a good strong 3 minutes with your favourite regex parser and 
you're good to go.

<https://www.thedave.ca/notes/?2c26ac9ad189da89#OKRTLrm9SoCUQORql2nYxg0iJa7lwXo/Xct+hXF5wwY=> 
has both the answer and shows my work.

But remember, this list is only valid until it isn't, even big providers 
move things around, sometimes frequently, so expect to update the list 
frequently (or again, don't, just use the tools that exist to do it in 
real time and go watch a movie instead).