You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2018/02/27 04:56:59 UTC

Please keep s3://spark-related-packages/ alive

If you go to the Downloads <http://spark.apache.org/downloads.html> page
and download Spark 2.2.1, you’ll get a link to an Apache mirror. It didn’t
use to be this way. As recently as Spark 2.2.0, downloads were served via
CloudFront <https://aws.amazon.com/cloudfront/>, which was backed by an S3
bucket named spark-related-packages.

It seems that we’ve stopped using CloudFront, and the S3 bucket behind it
has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m guessing
this is part of an effort to use the Apache mirror network, like other
Apache projects do.

From a user perspective, the Apache mirror network is several steps down
from using a modern CDN. Let me summarize why:

1. *Apache mirrors are often slow.* Apache does not impose any
performance requirements on its mirrors
<https://issues.apache.org/jira/browse/INFRA-10999?focusedCommentId=15717950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15717950>.
The difference between getting a good mirror and a bad one means
downloading Spark in less than a minute vs. 20 minutes. The problem is so
bad that I’ve thought about adding an Apache mirror blacklist
<https://github.com/nchammas/flintrock/issues/84#issuecomment-185038678>
to Flintrock to avoid getting one of these dud mirrors.
2. *Apache mirrors are inconvenient to use.* When you download something
from an Apache mirror, you get a link like this one
<https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
Instead of automatically redirecting you to your download, though, you need
to process the results you get back
<https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
to find your download target. And you need to handle the high download
failure rate, since sometimes the mirror you get doesn’t have the file it
claims to have.
3. *Apache mirrors are incomplete.* Apache mirrors only keep around the
latest releases, save for a few “archive” mirrors, which are often slow. So
if you want to download anything but the latest version of Spark, you are
out of luck.

Some of these problems can be mitigated by picking a specific mirror that
works well and hardcoding it in your scripts, but that defeats the purpose
of dynamically selecting a mirror and makes you a “bad” user of the mirror
network.

I raised some of these issues over on INFRA-10999
<https://issues.apache.org/jira/browse/INFRA-10999>. The ticket sat for a
year before I heard anything back, and the bottom line was that none of the
above problems have a solution on the horizon. It’s fine. I understand that
Apache is a volunteer organization and that the infrastructure team has a
lot to manage as it is. I still find it disappointing that an organization
of Apache’s stature doesn’t have a better solution for this in
collaboration with a third party. Python serves PyPI downloads using Fastly
<https://www.fastly.com/> and Homebrew serves packages using Bintray
<https://bintray.com/>. They both work really, really well. Why don’t we
have something as good for Apache projects? Anyway, that’s a separate
discussion.

What I want to say is this:

Dear whoever owns the spark-related-packages S3 bucket
<https://s3.amazonaws.com/spark-related-packages/>,

Please keep the bucket up-to-date with the latest Spark releases, alongside
the past releases that are already on there. It’s a huge help to the
Flintrock <https://github.com/nchammas/flintrock> project, and it’s an
equally big help to those of us writing infrastructure automation scripts
that deploy Spark in other contexts.

I understand that hosting this stuff is not free, and that I am not paying
anything for this service. If it needs to go, so be it. But I wanted to
take this opportunity to lay out the benefits I’ve enjoyed thanks to having
this bucket around, and to make sure that if it did die, it didn’t die a
quiet death.

Sincerely,
Nick

Re: Please keep s3://spark-related-packages/ alive

Posted by Reynold Xin <rx...@databricks.com>.

This was actually an AMPLab bucket.

On Feb 27, 2018, 6:04 PM +1300, Holden Karau <ho...@gmail.com>, wrote:
> Thanks Nick, we deprecated this during the roll over to the new release managers. I assume this bucket was maintained by someone at databricks so maybe they can chime in.
>
> > On Feb 26, 2018 8:57 PM, "Nicholas Chammas" <ni...@gmail.com> wrote:
> > > If you go to the Downloads page and download Spark 2.2.1, you’ll get a link to an Apache mirror. It didn’t use to be this way. As recently as Spark 2.2.0, downloads were served via CloudFront, which was backed by an S3 bucket named spark-related-packages.
> > > It seems that we’ve stopped using CloudFront, and the S3 bucket behind it has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m guessing this is part of an effort to use the Apache mirror network, like other Apache projects do.
> > > From a user perspective, the Apache mirror network is several steps down from using a modern CDN. Let me summarize why:
> > >
> > > 1. Apache mirrors are often slow. Apache does not impose any performance requirements on its mirrors. The difference between getting a good mirror and a bad one means downloading Spark in less than a minute vs. 20 minutes. The problem is so bad that I’ve thought about adding an Apache mirror blacklist to Flintrock to avoid getting one of these dud mirrors.
> > > 2. Apache mirrors are inconvenient to use. When you download something from an Apache mirror, you get a link like this one. Instead of automatically redirecting you to your download, though, you need to process the results you get back to find your download target. And you need to handle the high download failure rate, since sometimes the mirror you get doesn’t have the file it claims to have.
> > > 3. Apache mirrors are incomplete. Apache mirrors only keep around the latest releases, save for a few “archive” mirrors, which are often slow. So if you want to download anything but the latest version of Spark, you are out of luck.
> > >
> > > Some of these problems can be mitigated by picking a specific mirror that works well and hardcoding it in your scripts, but that defeats the purpose of dynamically selecting a mirror and makes you a “bad” user of the mirror network.
> > > I raised some of these issues over on INFRA-10999. The ticket sat for a year before I heard anything back, and the bottom line was that none of the above problems have a solution on the horizon. It’s fine. I understand that Apache is a volunteer organization and that the infrastructure team has a lot to manage as it is. I still find it disappointing that an organization of Apache’s stature doesn’t have a better solution for this in collaboration with a third party. Python serves PyPI downloads using Fastly and Homebrew serves packages using Bintray. They both work really, really well. Why don’t we have something as good for Apache projects? Anyway, that’s a separate discussion.
> > > What I want to say is this:
> > > Dear whoever owns the spark-related-packages S3 bucket,
> > > Please keep the bucket up-to-date with the latest Spark releases, alongside the past releases that are already on there. It’s a huge help to the Flintrock project, and it’s an equally big help to those of us writing infrastructure automation scripts that deploy Spark in other contexts.
> > > I understand that hosting this stuff is not free, and that I am not paying anything for this service. If it needs to go, so be it. But I wanted to take this opportunity to lay out the benefits I’ve enjoyed thanks to having this bucket around, and to make sure that if it did die, it didn’t die a quiet death.
> > > Sincerely,
> > > Nick
> > >
>

Re: Please keep s3://spark-related-packages/ alive

Posted by Holden Karau <ho...@gmail.com>.

Thanks Nick, we deprecated this during the roll over to the new release
managers. I assume this bucket was maintained by someone at databricks so
maybe they can chime in.

On Feb 26, 2018 8:57 PM, "Nicholas Chammas" <ni...@gmail.com>
wrote:

From a user perspective, the Apache mirror network is several steps down
from using a modern CDN. Let me summarize why:

What I want to say is this:

Dear whoever owns the spark-related-packages S3 bucket
<https://s3.amazonaws.com/spark-related-packages/>,

Sincerely,
Nick

Re: Please keep s3://spark-related-packages/ alive

Posted by Nicholas Chammas <ni...@gmail.com>.

Marton,

Thanks for the tip. (Too bad the docs
<https://reference.apache.org/pmc/mirror_scripts> referenced from the issue
I opened with INFRA <https://issues.apache.org/jira/browse/INFRA-10999> make
no mention of mirrors.cgi.)

Matei,

A Requester Pays bucket is a good idea. I was trying to avoid having to
maintain a repository of assets, but I suppose it's ultimately unavoidable
given that Apache does not partner with a CDN. I will look into this for
Flintrock.

Nick

On Wed, Feb 28, 2018 at 7:21 AM Marton, Elek <el...@apache.org> wrote:

>
> >  2. *Apache mirrors are inconvenient to use.* When you download
> >     something from an Apache mirror, you get a link like this one
> >     <
> https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
> >.
> >     Instead of automatically redirecting you to your download, though,
> >     you need to process the results you get back
> >     <
> https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42
> >
> >     to find your download target. And you need to handle the high
> >     download failure rate, since sometimes the mirror you get doesn’t
> >     have the file it claims to have.
>
> It's not a full answer, just a note:
>
> You can also use mirrors.cgi instead of parsing the json from closer.lua:
>
>
> https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
>
> (Unfortunatelly it doesn't check the availibility of the file. If it's
> moved to the archive you will be redirected to a 404)
>
> Marton
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: Please keep s3://spark-related-packages/ alive

Posted by "Marton, Elek" <el...@apache.org>.

>  2. *Apache mirrors are inconvenient to use.* When you download
>     something from an Apache mirror, you get a link like this one
>     <https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
>     Instead of automatically redirecting you to your download, though,
>     you need to process the results you get back
>     <https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
>     to find your download target. And you need to handle the high
>     download failure rate, since sometimes the mirror you get doesn’t
>     have the file it claims to have.

It's not a full answer, just a note:

You can also use mirrors.cgi instead of parsing the json from closer.lua:

https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz

(Unfortunatelly it doesn't check the availibility of the file. If it's 
moved to the archive you will be redirected to a 404)

Marton

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: Please keep s3://spark-related-packages/ alive

Posted by Matei Zaharia <ma...@gmail.com>.

For Flintrock, have you considered using a Requester Pays bucket? That way you’d get the availability of S3 without having to foot the bill for bandwidth yourself (which was the bulk of the cost for the old bucket).

Matei

> On Feb 27, 2018, at 4:35 PM, Nicholas Chammas <ni...@gmail.com> wrote:
> 
> So is there no hope for this S3 bucket, or room to replace it with a bucket owned by some organization other than AMPLab (which is technically now defunct, I guess)? Sorry to persist, but I just have to ask.
> 
> On Tue, Feb 27, 2018 at 10:36 AM Michael Heuer <he...@gmail.com> wrote:
> On Tue, Feb 27, 2018 at 8:17 AM, Sean Owen <sr...@gmail.com> wrote:
> See http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-d3kbcqa49mib13-cloudfront-net-td22427.html -- it was 'retired', yes.
> 
> Agree with all that, though they're intended for occasional individual use and not a case where performance and uptime matter. For that, I think you'd want to just host your own copy of the bits you need. 
> 
> The notional problem was that the S3 bucket wasn't obviously controlled/blessed by the ASF and yet was a source of official bits. It was another set of third-party credentials to hand around to release managers, which was IIRC a little problematic.
> 
> Homebrew does host distributions of ASF projects, like Spark, FWIW. 
> 
> To clarify, the apache-spark.rb formula in Homebrew uses the Apache mirror closer.lua script
> 
> https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-spark.rb#L4
> 
>    michael
> 
>  
> On Mon, Feb 26, 2018 at 10:57 PM Nicholas Chammas <ni...@gmail.com> wrote:
> If you go to the Downloads page and download Spark 2.2.1, you’ll get a link to an Apache mirror. It didn’t use to be this way. As recently as Spark 2.2.0, downloads were served via CloudFront, which was backed by an S3 bucket named spark-related-packages.
> 
> It seems that we’ve stopped using CloudFront, and the S3 bucket behind it has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m guessing this is part of an effort to use the Apache mirror network, like other Apache projects do.
> 
> From a user perspective, the Apache mirror network is several steps down from using a modern CDN. Let me summarize why:
> 
> 	• Apache mirrors are often slow. Apache does not impose any performance requirements on its mirrors. The difference between getting a good mirror and a bad one means downloading Spark in less than a minute vs. 20 minutes. The problem is so bad that I’ve thought about adding an Apache mirror blacklist to Flintrock to avoid getting one of these dud mirrors.
> 	• Apache mirrors are inconvenient to use. When you download something from an Apache mirror, you get a link like this one. Instead of automatically redirecting you to your download, though, you need to process the results you get back to find your download target. And you need to handle the high download failure rate, since sometimes the mirror you get doesn’t have the file it claims to have.
> 	• Apache mirrors are incomplete. Apache mirrors only keep around the latest releases, save for a few “archive” mirrors, which are often slow. So if you want to download anything but the latest version of Spark, you are out of luck.
> Some of these problems can be mitigated by picking a specific mirror that works well and hardcoding it in your scripts, but that defeats the purpose of dynamically selecting a mirror and makes you a “bad” user of the mirror network.
> 
> I raised some of these issues over on INFRA-10999. The ticket sat for a year before I heard anything back, and the bottom line was that none of the above problems have a solution on the horizon. It’s fine. I understand that Apache is a volunteer organization and that the infrastructure team has a lot to manage as it is. I still find it disappointing that an organization of Apache’s stature doesn’t have a better solution for this in collaboration with a third party. Python serves PyPI downloads using Fastly and Homebrew serves packages using Bintray. They both work really, really well. Why don’t we have something as good for Apache projects? Anyway, that’s a separate discussion.
> 
> What I want to say is this:
> 
> Dear whoever owns the spark-related-packages S3 bucket,
> 
> Please keep the bucket up-to-date with the latest Spark releases, alongside the past releases that are already on there. It’s a huge help to the Flintrock project, and it’s an equally big help to those of us writing infrastructure automation scripts that deploy Spark in other contexts.
> 
> I understand that hosting this stuff is not free, and that I am not paying anything for this service. If it needs to go, so be it. But I wanted to take this opportunity to lay out the benefits I’ve enjoyed thanks to having this bucket around, and to make sure that if it did die, it didn’t die a quiet death.
> 
> Sincerely,
> Nick
> 


---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: Please keep s3://spark-related-packages/ alive

Posted by Nicholas Chammas <ni...@gmail.com>.

So is there no hope for this S3 bucket, or room to replace it with a bucket
owned by some organization other than AMPLab (which is technically now
defunct <https://amplab.cs.berkeley.edu/endofproject/>, I guess)? Sorry to
persist, but I just have to ask.

On Tue, Feb 27, 2018 at 10:36 AM Michael Heuer <he...@gmail.com> wrote:

> On Tue, Feb 27, 2018 at 8:17 AM, Sean Owen <sr...@gmail.com> wrote:
>
>> See
>> http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-d3kbcqa49mib13-cloudfront-net-td22427.html --
>> it was 'retired', yes.
>>
>> Agree with all that, though they're intended for occasional individual
>> use and not a case where performance and uptime matter. For that, I think
>> you'd want to just host your own copy of the bits you need.
>>
>> The notional problem was that the S3 bucket wasn't obviously
>> controlled/blessed by the ASF and yet was a source of official bits. It was
>> another set of third-party credentials to hand around to release managers,
>> which was IIRC a little problematic.
>>
>> Homebrew does host distributions of ASF projects, like Spark, FWIW.
>>
>
> To clarify, the apache-spark.rb formula in Homebrew uses the Apache
> mirror closer.lua script
>
>
> https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-spark.rb#L4
>
>    michael
>
>
>
>> On Mon, Feb 26, 2018 at 10:57 PM Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> If you go to the Downloads <http://spark.apache.org/downloads.html>
>>> page and download Spark 2.2.1, you’ll get a link to an Apache mirror. It
>>> didn’t use to be this way. As recently as Spark 2.2.0, downloads were
>>> served via CloudFront <https://aws.amazon.com/cloudfront/>, which was
>>> backed by an S3 bucket named spark-related-packages.
>>>
>>> It seems that we’ve stopped using CloudFront, and the S3 bucket behind
>>> it has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m
>>> guessing this is part of an effort to use the Apache mirror network, like
>>> other Apache projects do.
>>>
>>> From a user perspective, the Apache mirror network is several steps down
>>> from using a modern CDN. Let me summarize why:
>>>
>>>    1. *Apache mirrors are often slow.* Apache does not impose any
>>>    performance requirements on its mirrors
>>>    <https://issues.apache.org/jira/browse/INFRA-10999?focusedCommentId=15717950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15717950>.
>>>    The difference between getting a good mirror and a bad one means
>>>    downloading Spark in less than a minute vs. 20 minutes. The problem is so
>>>    bad that I’ve thought about adding an Apache mirror blacklist
>>>    <https://github.com/nchammas/flintrock/issues/84#issuecomment-185038678>
>>>    to Flintrock to avoid getting one of these dud mirrors.
>>>    2. *Apache mirrors are inconvenient to use.* When you download
>>>    something from an Apache mirror, you get a link like this one
>>>    <https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
>>>    Instead of automatically redirecting you to your download, though, you need
>>>    to process the results you get back
>>>    <https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
>>>    to find your download target. And you need to handle the high download
>>>    failure rate, since sometimes the mirror you get doesn’t have the file it
>>>    claims to have.
>>>    3. *Apache mirrors are incomplete.* Apache mirrors only keep around
>>>    the latest releases, save for a few “archive” mirrors, which are often
>>>    slow. So if you want to download anything but the latest version of Spark,
>>>    you are out of luck.
>>>
>>> Some of these problems can be mitigated by picking a specific mirror
>>> that works well and hardcoding it in your scripts, but that defeats the
>>> purpose of dynamically selecting a mirror and makes you a “bad” user of the
>>> mirror network.
>>>
>>> I raised some of these issues over on INFRA-10999
>>> <https://issues.apache.org/jira/browse/INFRA-10999>. The ticket sat for
>>> a year before I heard anything back, and the bottom line was that none of
>>> the above problems have a solution on the horizon. It’s fine. I understand
>>> that Apache is a volunteer organization and that the infrastructure team
>>> has a lot to manage as it is. I still find it disappointing that an
>>> organization of Apache’s stature doesn’t have a better solution for this in
>>> collaboration with a third party. Python serves PyPI downloads using
>>> Fastly <https://www.fastly.com/> and Homebrew serves packages using
>>> Bintray <https://bintray.com/>. They both work really, really well. Why
>>> don’t we have something as good for Apache projects? Anyway, that’s a
>>> separate discussion.
>>>
>>> What I want to say is this:
>>>
>>> Dear whoever owns the spark-related-packages S3 bucket
>>> <https://s3.amazonaws.com/spark-related-packages/>,
>>>
>>> Please keep the bucket up-to-date with the latest Spark releases,
>>> alongside the past releases that are already on there. It’s a huge help to
>>> the Flintrock <https://github.com/nchammas/flintrock> project, and it’s
>>> an equally big help to those of us writing infrastructure automation
>>> scripts that deploy Spark in other contexts.
>>>
>>> I understand that hosting this stuff is not free, and that I am not
>>> paying anything for this service. If it needs to go, so be it. But I wanted
>>> to take this opportunity to lay out the benefits I’ve enjoyed thanks to
>>> having this bucket around, and to make sure that if it did die, it didn’t
>>> die a quiet death.
>>>
>>> Sincerely,
>>> Nick
>>> 
>>>
>>

Re: Please keep s3://spark-related-packages/ alive

Posted by Michael Heuer <he...@gmail.com>.

On Tue, Feb 27, 2018 at 8:17 AM, Sean Owen <sr...@gmail.com> wrote:

> See http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-
> d3kbcqa49mib13-cloudfront-net-td22427.html -- it was 'retired', yes.
>
> Agree with all that, though they're intended for occasional individual use
> and not a case where performance and uptime matter. For that, I think you'd
> want to just host your own copy of the bits you need.
>
> The notional problem was that the S3 bucket wasn't obviously
> controlled/blessed by the ASF and yet was a source of official bits. It was
> another set of third-party credentials to hand around to release managers,
> which was IIRC a little problematic.
>
> Homebrew does host distributions of ASF projects, like Spark, FWIW.
>

To clarify, the apache-spark.rb formula in Homebrew uses the Apache mirror
closer.lua script

https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-spark.rb#L4

   michael



> On Mon, Feb 26, 2018 at 10:57 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> If you go to the Downloads <http://spark.apache.org/downloads.html> page
>> and download Spark 2.2.1, you’ll get a link to an Apache mirror. It didn’t
>> use to be this way. As recently as Spark 2.2.0, downloads were served via
>> CloudFront <https://aws.amazon.com/cloudfront/>, which was backed by an
>> S3 bucket named spark-related-packages.
>>
>> It seems that we’ve stopped using CloudFront, and the S3 bucket behind it
>> has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m guessing
>> this is part of an effort to use the Apache mirror network, like other
>> Apache projects do.
>>
>> From a user perspective, the Apache mirror network is several steps down
>> from using a modern CDN. Let me summarize why:
>>
>>    1. *Apache mirrors are often slow.* Apache does not impose any
>>    performance requirements on its mirrors
>>    <https://issues.apache.org/jira/browse/INFRA-10999?focusedCommentId=15717950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15717950>.
>>    The difference between getting a good mirror and a bad one means
>>    downloading Spark in less than a minute vs. 20 minutes. The problem is so
>>    bad that I’ve thought about adding an Apache mirror blacklist
>>    <https://github.com/nchammas/flintrock/issues/84#issuecomment-185038678>
>>    to Flintrock to avoid getting one of these dud mirrors.
>>    2. *Apache mirrors are inconvenient to use.* When you download
>>    something from an Apache mirror, you get a link like this one
>>    <https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
>>    Instead of automatically redirecting you to your download, though, you need
>>    to process the results you get back
>>    <https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
>>    to find your download target. And you need to handle the high download
>>    failure rate, since sometimes the mirror you get doesn’t have the file it
>>    claims to have.
>>    3. *Apache mirrors are incomplete.* Apache mirrors only keep around
>>    the latest releases, save for a few “archive” mirrors, which are often
>>    slow. So if you want to download anything but the latest version of Spark,
>>    you are out of luck.
>>
>> Some of these problems can be mitigated by picking a specific mirror that
>> works well and hardcoding it in your scripts, but that defeats the purpose
>> of dynamically selecting a mirror and makes you a “bad” user of the mirror
>> network.
>>
>> I raised some of these issues over on INFRA-10999
>> <https://issues.apache.org/jira/browse/INFRA-10999>. The ticket sat for
>> a year before I heard anything back, and the bottom line was that none of
>> the above problems have a solution on the horizon. It’s fine. I understand
>> that Apache is a volunteer organization and that the infrastructure team
>> has a lot to manage as it is. I still find it disappointing that an
>> organization of Apache’s stature doesn’t have a better solution for this in
>> collaboration with a third party. Python serves PyPI downloads using
>> Fastly <https://www.fastly.com/> and Homebrew serves packages using
>> Bintray <https://bintray.com/>. They both work really, really well. Why
>> don’t we have something as good for Apache projects? Anyway, that’s a
>> separate discussion.
>>
>> What I want to say is this:
>>
>> Dear whoever owns the spark-related-packages S3 bucket
>> <https://s3.amazonaws.com/spark-related-packages/>,
>>
>> Please keep the bucket up-to-date with the latest Spark releases,
>> alongside the past releases that are already on there. It’s a huge help to
>> the Flintrock <https://github.com/nchammas/flintrock> project, and it’s
>> an equally big help to those of us writing infrastructure automation
>> scripts that deploy Spark in other contexts.
>>
>> I understand that hosting this stuff is not free, and that I am not
>> paying anything for this service. If it needs to go, so be it. But I wanted
>> to take this opportunity to lay out the benefits I’ve enjoyed thanks to
>> having this bucket around, and to make sure that if it did die, it didn’t
>> die a quiet death.
>>
>> Sincerely,
>> Nick
>> 
>>
>

Re: Please keep s3://spark-related-packages/ alive

Posted by Sean Owen <sr...@gmail.com>.

See
http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-d3kbcqa49mib13-cloudfront-net-td22427.html
--
it was 'retired', yes.

Agree with all that, though they're intended for occasional individual use
and not a case where performance and uptime matter. For that, I think you'd
want to just host your own copy of the bits you need.

The notional problem was that the S3 bucket wasn't obviously
controlled/blessed by the ASF and yet was a source of official bits. It was
another set of third-party credentials to hand around to release managers,
which was IIRC a little problematic.

Homebrew does host distributions of ASF projects, like Spark, FWIW.

On Mon, Feb 26, 2018 at 10:57 PM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> If you go to the Downloads <http://spark.apache.org/downloads.html> page
> and download Spark 2.2.1, you’ll get a link to an Apache mirror. It didn’t
> use to be this way. As recently as Spark 2.2.0, downloads were served via
> CloudFront <https://aws.amazon.com/cloudfront/>, which was backed by an
> S3 bucket named spark-related-packages.
>
> It seems that we’ve stopped using CloudFront, and the S3 bucket behind it
> has stopped receiving updates (e.g. Spark 2.2.1 isn’t there). I’m guessing
> this is part of an effort to use the Apache mirror network, like other
> Apache projects do.
>
> From a user perspective, the Apache mirror network is several steps down
> from using a modern CDN. Let me summarize why:
>
>    1. *Apache mirrors are often slow.* Apache does not impose any
>    performance requirements on its mirrors
>    <https://issues.apache.org/jira/browse/INFRA-10999?focusedCommentId=15717950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15717950>.
>    The difference between getting a good mirror and a bad one means
>    downloading Spark in less than a minute vs. 20 minutes. The problem is so
>    bad that I’ve thought about adding an Apache mirror blacklist
>    <https://github.com/nchammas/flintrock/issues/84#issuecomment-185038678>
>    to Flintrock to avoid getting one of these dud mirrors.
>    2. *Apache mirrors are inconvenient to use.* When you download
>    something from an Apache mirror, you get a link like this one
>    <https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz>.
>    Instead of automatically redirecting you to your download, though, you need
>    to process the results you get back
>    <https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42>
>    to find your download target. And you need to handle the high download
>    failure rate, since sometimes the mirror you get doesn’t have the file it
>    claims to have.
>    3. *Apache mirrors are incomplete.* Apache mirrors only keep around
>    the latest releases, save for a few “archive” mirrors, which are often
>    slow. So if you want to download anything but the latest version of Spark,
>    you are out of luck.
>
> Some of these problems can be mitigated by picking a specific mirror that
> works well and hardcoding it in your scripts, but that defeats the purpose
> of dynamically selecting a mirror and makes you a “bad” user of the mirror
> network.
>
> I raised some of these issues over on INFRA-10999
> <https://issues.apache.org/jira/browse/INFRA-10999>. The ticket sat for a
> year before I heard anything back, and the bottom line was that none of the
> above problems have a solution on the horizon. It’s fine. I understand that
> Apache is a volunteer organization and that the infrastructure team has a
> lot to manage as it is. I still find it disappointing that an organization
> of Apache’s stature doesn’t have a better solution for this in
> collaboration with a third party. Python serves PyPI downloads using
> Fastly <https://www.fastly.com/> and Homebrew serves packages using
> Bintray <https://bintray.com/>. They both work really, really well. Why
> don’t we have something as good for Apache projects? Anyway, that’s a
> separate discussion.
>
> What I want to say is this:
>
> Dear whoever owns the spark-related-packages S3 bucket
> <https://s3.amazonaws.com/spark-related-packages/>,
>
> Please keep the bucket up-to-date with the latest Spark releases,
> alongside the past releases that are already on there. It’s a huge help to
> the Flintrock <https://github.com/nchammas/flintrock> project, and it’s
> an equally big help to those of us writing infrastructure automation
> scripts that deploy Spark in other contexts.
>
> I understand that hosting this stuff is not free, and that I am not paying
> anything for this service. If it needs to go, so be it. But I wanted to
> take this opportunity to lay out the benefits I’ve enjoyed thanks to having
> this bucket around, and to make sure that if it did die, it didn’t die a
> quiet death.
>
> Sincerely,
> Nick
> 
>