You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/10/05 02:17:09 UTC
Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
I’m looking here:
https://s3.amazonaws.com/spark-related-packages/
I believe this is where one set of official packages is published. Please
correct me if this is not the case.
It appears that almost every version of Spark up to and including 1.5.0 has
included a --bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
However, 1.5.1 has no such package. There is a
spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
thing. (1.5.0 also has a hadoop1-scala2.11 package.)
Was this intentional?
More importantly, is there some rough specification for what packages we
should be able to expect in this S3 bucket with every release?
This is important for those of us who depend on this publishing venue (e.g.
spark-ec2 and related tools).
Nick
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Sean Owen <so...@cloudera.com>.
This is about the s3.amazonaws.com files, not dist.apache.org right?
or does it affect both?
(BTW you can keep as many old release artifacts around on the
apache.org archives as you like; I think the suggestion is to remove
all but the most recent releases from the set that's replicated to all
the Apache mirrors.)
On Wed, Oct 7, 2015 at 6:57 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I don't think we have a firm contract around that. So far we've never
> removed old artifacts, but the ASF has asked us at time to decrease the size
> of binaries we post. In the future at some point we may drop older ones
> since we keep adding new ones.
>
> If downstream projects are depending on our artifacts, I'd say just hold
> tight for now until something changes. If it changes, then those projects
> might need to build Spark on their own and host older hadoop versions, etc.
>
> On Wed, Oct 7, 2015 at 9:59 AM, Nicholas Chammas
> <ni...@gmail.com> wrote:
>>
>> Thanks guys.
>>
>> Regarding this earlier question:
>>
>> More importantly, is there some rough specification for what packages we
>> should be able to expect in this S3 bucket with every release?
>>
>> Is the implied answer that we should continue to expect the same set of
>> artifacts for every release for the foreseeable future?
>>
>> Nick
>>
>>
>> On Tue, Oct 6, 2015 at 1:13 AM Patrick Wendell <pw...@gmail.com> wrote:
>>>
>>> The missing artifacts are uploaded now. Things should propagate in the
>>> next 24 hours. If there are still issues past then ping this thread. Thanks!
>>>
>>> - Patrick
>>>
>>> On Mon, Oct 5, 2015 at 2:41 PM, Nicholas Chammas
>>> <ni...@gmail.com> wrote:
>>>>
>>>> Thanks for looking into this Josh.
>>>>
>>>> On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com>
>>>> wrote:
>>>>>
>>>>> I'm working on a fix for this right now. I'm planning to re-run a
>>>>> modified copy of the release packaging scripts which will emit only the
>>>>> missing artifacts (so we won't upload new artifacts with different SHAs for
>>>>> the builds which did succeed).
>>>>>
>>>>> I expect to have this finished in the next day or so; I'm currently
>>>>> blocked by some infra downtime but expect that to be resolved soon.
>>>>>
>>>>> - Josh
>>>>>
>>>>> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas
>>>>> <ni...@gmail.com> wrote:
>>>>>>
>>>>>> Blaž said:
>>>>>>
>>>>>> Also missing is
>>>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>>>> which breaks spark-ec2 script.
>>>>>>
>>>>>> This is the package I am referring to in my original email.
>>>>>>
>>>>>> Nick said:
>>>>>>
>>>>>> It appears that almost every version of Spark up to and including
>>>>>> 1.5.0 has included a —bin-hadoop1.tgz release (e.g.
>>>>>> spark-1.5.0-bin-hadoop1.tgz). However, 1.5.1 has no such package.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>>>>>>>
>>>>>>> Also missing is
>>>>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>>>>> which breaks spark-ec2 script.
>>>>>>>
>>>>>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>>>>>>
>>>>>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>>>>>
>>>>>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas
>>>>>>>> <ni...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I’m looking here:
>>>>>>>>>
>>>>>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>>>>>
>>>>>>>>> I believe this is where one set of official packages is published.
>>>>>>>>> Please correct me if this is not the case.
>>>>>>>>>
>>>>>>>>> It appears that almost every version of Spark up to and including
>>>>>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>>>>>
>>>>>>>>> However, 1.5.1 has no such package. There is a
>>>>>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate thing.
>>>>>>>>> (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>>>>>
>>>>>>>>> Was this intentional?
>>>>>>>>>
>>>>>>>>> More importantly, is there some rough specification for what
>>>>>>>>> packages we should be able to expect in this S3 bucket with every release?
>>>>>>>>>
>>>>>>>>> This is important for those of us who depend on this publishing
>>>>>>>>> venue (e.g. spark-ec2 and related tools).
>>>>>>>>>
>>>>>>>>> Nick
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Nicholas Chammas <ni...@gmail.com>.
Sounds good to me.
For my purposes, I'm less concerned about old Spark artifacts and more
concerned about the consistency of the set of artifacts that get generated
with new releases. (e.g. Each new release will always include one artifact
each for Hadoop 1, Hadoop 1 + Scala 2.11, etc...)
It sounds like we can expect that set to stay the same with new releases
for now, but it's not a hard guarantee. I think that's fine for now.
Nick
On Wed, Oct 7, 2015 at 1:57 PM Patrick Wendell <pw...@gmail.com> wrote:
> I don't think we have a firm contract around that. So far we've never
> removed old artifacts, but the ASF has asked us at time to decrease the
> size of binaries we post. In the future at some point we may drop older
> ones since we keep adding new ones.
>
> If downstream projects are depending on our artifacts, I'd say just hold
> tight for now until something changes. If it changes, then those projects
> might need to build Spark on their own and host older hadoop versions, etc.
>
> On Wed, Oct 7, 2015 at 9:59 AM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Thanks guys.
>>
>> Regarding this earlier question:
>>
>> More importantly, is there some rough specification for what packages we
>> should be able to expect in this S3 bucket with every release?
>>
>> Is the implied answer that we should continue to expect the same set of
>> artifacts for every release for the foreseeable future?
>>
>> Nick
>>
>>
>> On Tue, Oct 6, 2015 at 1:13 AM Patrick Wendell <pw...@gmail.com>
>> wrote:
>>
>>> The missing artifacts are uploaded now. Things should propagate in the
>>> next 24 hours. If there are still issues past then ping this thread. Thanks!
>>>
>>> - Patrick
>>>
>>> On Mon, Oct 5, 2015 at 2:41 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Thanks for looking into this Josh.
>>>>
>>>> On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com>
>>>> wrote:
>>>>
>>>>> I'm working on a fix for this right now. I'm planning to re-run a
>>>>> modified copy of the release packaging scripts which will emit only the
>>>>> missing artifacts (so we won't upload new artifacts with different SHAs for
>>>>> the builds which *did* succeed).
>>>>>
>>>>> I expect to have this finished in the next day or so; I'm currently
>>>>> blocked by some infra downtime but expect that to be resolved soon.
>>>>>
>>>>> - Josh
>>>>>
>>>>> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <
>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>
>>>>>> Blaž said:
>>>>>>
>>>>>> Also missing is
>>>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>>>> which breaks spark-ec2 script.
>>>>>>
>>>>>> This is the package I am referring to in my original email.
>>>>>>
>>>>>> Nick said:
>>>>>>
>>>>>> It appears that almost every version of Spark up to and including
>>>>>> 1.5.0 has included a —bin-hadoop1.tgz release (e.g.
>>>>>> spark-1.5.0-bin-hadoop1.tgz). However, 1.5.1 has no such package.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Also missing is
>>>>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>>>>> which breaks spark-ec2 script.
>>>>>>>
>>>>>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>>>>>>
>>>>>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>>>>>
>>>>>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>>>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I’m looking here:
>>>>>>>>>
>>>>>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>>>>>
>>>>>>>>> I believe this is where one set of official packages is published.
>>>>>>>>> Please correct me if this is not the case.
>>>>>>>>>
>>>>>>>>> It appears that almost every version of Spark up to and including
>>>>>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>>>>>
>>>>>>>>> However, 1.5.1 has no such package. There is a
>>>>>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a
>>>>>>>>> separate thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>>>>>
>>>>>>>>> Was this intentional?
>>>>>>>>>
>>>>>>>>> More importantly, is there some rough specification for what
>>>>>>>>> packages we should be able to expect in this S3 bucket with every release?
>>>>>>>>>
>>>>>>>>> This is important for those of us who depend on this publishing
>>>>>>>>> venue (e.g. spark-ec2 and related tools).
>>>>>>>>>
>>>>>>>>> Nick
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Patrick Wendell <pw...@gmail.com>.
I don't think we have a firm contract around that. So far we've never
removed old artifacts, but the ASF has asked us at time to decrease the
size of binaries we post. In the future at some point we may drop older
ones since we keep adding new ones.
If downstream projects are depending on our artifacts, I'd say just hold
tight for now until something changes. If it changes, then those projects
might need to build Spark on their own and host older hadoop versions, etc.
On Wed, Oct 7, 2015 at 9:59 AM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> Thanks guys.
>
> Regarding this earlier question:
>
> More importantly, is there some rough specification for what packages we
> should be able to expect in this S3 bucket with every release?
>
> Is the implied answer that we should continue to expect the same set of
> artifacts for every release for the foreseeable future?
>
> Nick
>
>
> On Tue, Oct 6, 2015 at 1:13 AM Patrick Wendell <pw...@gmail.com> wrote:
>
>> The missing artifacts are uploaded now. Things should propagate in the
>> next 24 hours. If there are still issues past then ping this thread. Thanks!
>>
>> - Patrick
>>
>> On Mon, Oct 5, 2015 at 2:41 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Thanks for looking into this Josh.
>>>
>>> On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com>
>>> wrote:
>>>
>>>> I'm working on a fix for this right now. I'm planning to re-run a
>>>> modified copy of the release packaging scripts which will emit only the
>>>> missing artifacts (so we won't upload new artifacts with different SHAs for
>>>> the builds which *did* succeed).
>>>>
>>>> I expect to have this finished in the next day or so; I'm currently
>>>> blocked by some infra downtime but expect that to be resolved soon.
>>>>
>>>> - Josh
>>>>
>>>> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> Blaž said:
>>>>>
>>>>> Also missing is
>>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>>> which breaks spark-ec2 script.
>>>>>
>>>>> This is the package I am referring to in my original email.
>>>>>
>>>>> Nick said:
>>>>>
>>>>> It appears that almost every version of Spark up to and including
>>>>> 1.5.0 has included a —bin-hadoop1.tgz release (e.g.
>>>>> spark-1.5.0-bin-hadoop1.tgz). However, 1.5.1 has no such package.
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>>>>>
>>>>>> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-
>>>>>> 1.5.1-bin-hadoop1.tgz which breaks spark-ec2 script.
>>>>>>
>>>>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>>>>>
>>>>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>>>>
>>>>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>>>
>>>>>>>> I’m looking here:
>>>>>>>>
>>>>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>>>>
>>>>>>>> I believe this is where one set of official packages is published.
>>>>>>>> Please correct me if this is not the case.
>>>>>>>>
>>>>>>>> It appears that almost every version of Spark up to and including
>>>>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>>>>
>>>>>>>> However, 1.5.1 has no such package. There is a
>>>>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a
>>>>>>>> separate thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>>>>
>>>>>>>> Was this intentional?
>>>>>>>>
>>>>>>>> More importantly, is there some rough specification for what
>>>>>>>> packages we should be able to expect in this S3 bucket with every release?
>>>>>>>>
>>>>>>>> This is important for those of us who depend on this publishing
>>>>>>>> venue (e.g. spark-ec2 and related tools).
>>>>>>>>
>>>>>>>> Nick
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Nicholas Chammas <ni...@gmail.com>.
Thanks guys.
Regarding this earlier question:
More importantly, is there some rough specification for what packages we
should be able to expect in this S3 bucket with every release?
Is the implied answer that we should continue to expect the same set of
artifacts for every release for the foreseeable future?
Nick
On Tue, Oct 6, 2015 at 1:13 AM Patrick Wendell <pw...@gmail.com> wrote:
> The missing artifacts are uploaded now. Things should propagate in the
> next 24 hours. If there are still issues past then ping this thread. Thanks!
>
> - Patrick
>
> On Mon, Oct 5, 2015 at 2:41 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Thanks for looking into this Josh.
>>
>> On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com>
>> wrote:
>>
>>> I'm working on a fix for this right now. I'm planning to re-run a
>>> modified copy of the release packaging scripts which will emit only the
>>> missing artifacts (so we won't upload new artifacts with different SHAs for
>>> the builds which *did* succeed).
>>>
>>> I expect to have this finished in the next day or so; I'm currently
>>> blocked by some infra downtime but expect that to be resolved soon.
>>>
>>> - Josh
>>>
>>> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Blaž said:
>>>>
>>>> Also missing is
>>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>>> which breaks spark-ec2 script.
>>>>
>>>> This is the package I am referring to in my original email.
>>>>
>>>> Nick said:
>>>>
>>>> It appears that almost every version of Spark up to and including 1.5.0
>>>> has included a —bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
>>>> However, 1.5.1 has no such package.
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>>>>
>>>>> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-
>>>>> 1.5.1-bin-hadoop1.tgz which breaks spark-ec2 script.
>>>>>
>>>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>>>
>>>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>>
>>>>>>> I’m looking here:
>>>>>>>
>>>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>>>
>>>>>>> I believe this is where one set of official packages is published.
>>>>>>> Please correct me if this is not the case.
>>>>>>>
>>>>>>> It appears that almost every version of Spark up to and including
>>>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>>>
>>>>>>> However, 1.5.1 has no such package. There is a
>>>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a
>>>>>>> separate thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>>>
>>>>>>> Was this intentional?
>>>>>>>
>>>>>>> More importantly, is there some rough specification for what
>>>>>>> packages we should be able to expect in this S3 bucket with every release?
>>>>>>>
>>>>>>> This is important for those of us who depend on this publishing
>>>>>>> venue (e.g. spark-ec2 and related tools).
>>>>>>>
>>>>>>> Nick
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Patrick Wendell <pw...@gmail.com>.
The missing artifacts are uploaded now. Things should propagate in the next
24 hours. If there are still issues past then ping this thread. Thanks!
- Patrick
On Mon, Oct 5, 2015 at 2:41 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> Thanks for looking into this Josh.
>
> On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com>
> wrote:
>
>> I'm working on a fix for this right now. I'm planning to re-run a
>> modified copy of the release packaging scripts which will emit only the
>> missing artifacts (so we won't upload new artifacts with different SHAs for
>> the builds which *did* succeed).
>>
>> I expect to have this finished in the next day or so; I'm currently
>> blocked by some infra downtime but expect that to be resolved soon.
>>
>> - Josh
>>
>> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Blaž said:
>>>
>>> Also missing is
>>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>>> which breaks spark-ec2 script.
>>>
>>> This is the package I am referring to in my original email.
>>>
>>> Nick said:
>>>
>>> It appears that almost every version of Spark up to and including 1.5.0
>>> has included a —bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
>>> However, 1.5.1 has no such package.
>>>
>>> Nick
>>>
>>>
>>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>>>
>>>> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-
>>>> 1.5.1-bin-hadoop1.tgz which breaks spark-ec2 script.
>>>>
>>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>>
>>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>
>>>>>> I’m looking here:
>>>>>>
>>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>>
>>>>>> I believe this is where one set of official packages is published.
>>>>>> Please correct me if this is not the case.
>>>>>>
>>>>>> It appears that almost every version of Spark up to and including
>>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>>
>>>>>> However, 1.5.1 has no such package. There is a
>>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a
>>>>>> separate thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>>
>>>>>> Was this intentional?
>>>>>>
>>>>>> More importantly, is there some rough specification for what packages
>>>>>> we should be able to expect in this S3 bucket with every release?
>>>>>>
>>>>>> This is important for those of us who depend on this publishing venue
>>>>>> (e.g. spark-ec2 and related tools).
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Nicholas Chammas <ni...@gmail.com>.
Thanks for looking into this Josh.
On Mon, Oct 5, 2015 at 5:39 PM Josh Rosen <jo...@databricks.com> wrote:
> I'm working on a fix for this right now. I'm planning to re-run a modified
> copy of the release packaging scripts which will emit only the missing
> artifacts (so we won't upload new artifacts with different SHAs for the
> builds which *did* succeed).
>
> I expect to have this finished in the next day or so; I'm currently
> blocked by some infra downtime but expect that to be resolved soon.
>
> - Josh
>
> On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Blaž said:
>>
>> Also missing is
>> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
>> which breaks spark-ec2 script.
>>
>> This is the package I am referring to in my original email.
>>
>> Nick said:
>>
>> It appears that almost every version of Spark up to and including 1.5.0
>> has included a —bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
>> However, 1.5.1 has no such package.
>>
>> Nick
>>
>>
>> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>>
>>> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-
>>> 1.5.1-bin-hadoop1.tgz which breaks spark-ec2 script.
>>>
>>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>>
>>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> I’m looking here:
>>>>>
>>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>>
>>>>> I believe this is where one set of official packages is published.
>>>>> Please correct me if this is not the case.
>>>>>
>>>>> It appears that almost every version of Spark up to and including
>>>>> 1.5.0 has included a --bin-hadoop1.tgz release (e.g.
>>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>>
>>>>> However, 1.5.1 has no such package. There is a
>>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
>>>>> thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>>
>>>>> Was this intentional?
>>>>>
>>>>> More importantly, is there some rough specification for what packages
>>>>> we should be able to expect in this S3 bucket with every release?
>>>>>
>>>>> This is important for those of us who depend on this publishing venue
>>>>> (e.g. spark-ec2 and related tools).
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>
>>>>
>>>
>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Josh Rosen <jo...@databricks.com>.
I'm working on a fix for this right now. I'm planning to re-run a modified
copy of the release packaging scripts which will emit only the missing
artifacts (so we won't upload new artifacts with different SHAs for the
builds which *did* succeed).
I expect to have this finished in the next day or so; I'm currently blocked
by some infra downtime but expect that to be resolved soon.
- Josh
On Mon, Oct 5, 2015 at 8:46 AM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> Blaž said:
>
> Also missing is
> http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
> which breaks spark-ec2 script.
>
> This is the package I am referring to in my original email.
>
> Nick said:
>
> It appears that almost every version of Spark up to and including 1.5.0
> has included a —bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
> However, 1.5.1 has no such package.
>
> Nick
>
>
> On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
>
>> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-
>> 1.5.1-bin-hadoop1.tgz which breaks spark-ec2 script.
>>
>> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>>
>>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> I’m looking here:
>>>>
>>>> https://s3.amazonaws.com/spark-related-packages/
>>>>
>>>> I believe this is where one set of official packages is published.
>>>> Please correct me if this is not the case.
>>>>
>>>> It appears that almost every version of Spark up to and including 1.5.0
>>>> has included a --bin-hadoop1.tgz release (e.g.
>>>> spark-1.5.0-bin-hadoop1.tgz).
>>>>
>>>> However, 1.5.1 has no such package. There is a
>>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
>>>> thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>>
>>>> Was this intentional?
>>>>
>>>> More importantly, is there some rough specification for what packages
>>>> we should be able to expect in this S3 bucket with every release?
>>>>
>>>> This is important for those of us who depend on this publishing venue
>>>> (e.g. spark-ec2 and related tools).
>>>>
>>>> Nick
>>>>
>>>>
>>>
>>>
>>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Nicholas Chammas <ni...@gmail.com>.
Blaž said:
Also missing is
http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
which breaks spark-ec2 script.
This is the package I am referring to in my original email.
Nick said:
It appears that almost every version of Spark up to and including 1.5.0 has
included a —bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz).
However, 1.5.1 has no such package.
Nick
On Mon, Oct 5, 2015 at 3:27 AM Blaž Šnuderl <sn...@gmail.com> wrote:
> Also missing is http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
> which breaks spark-ec2 script.
>
> On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> hadoop1 package for Scala 2.10 wasn't in RC1 either:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>>
>> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> I’m looking here:
>>>
>>> https://s3.amazonaws.com/spark-related-packages/
>>>
>>> I believe this is where one set of official packages is published.
>>> Please correct me if this is not the case.
>>>
>>> It appears that almost every version of Spark up to and including 1.5.0
>>> has included a --bin-hadoop1.tgz release (e.g.
>>> spark-1.5.0-bin-hadoop1.tgz).
>>>
>>> However, 1.5.1 has no such package. There is a
>>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
>>> thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>>
>>> Was this intentional?
>>>
>>> More importantly, is there some rough specification for what packages we
>>> should be able to expect in this S3 bucket with every release?
>>>
>>> This is important for those of us who depend on this publishing venue
>>> (e.g. spark-ec2 and related tools).
>>>
>>> Nick
>>>
>>>
>>
>>
>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Blaž Šnuderl <sn...@gmail.com>.
Also missing is
http://s3.amazonaws.com/spark-related-packages/spark-1.5.1-bin-hadoop1.tgz
which breaks spark-ec2 script.
On Mon, Oct 5, 2015 at 5:20 AM, Ted Yu <yu...@gmail.com> wrote:
> hadoop1 package for Scala 2.10 wasn't in RC1 either:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
>
> On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I’m looking here:
>>
>> https://s3.amazonaws.com/spark-related-packages/
>>
>> I believe this is where one set of official packages is published. Please
>> correct me if this is not the case.
>>
>> It appears that almost every version of Spark up to and including 1.5.0
>> has included a --bin-hadoop1.tgz release (e.g.
>> spark-1.5.0-bin-hadoop1.tgz).
>>
>> However, 1.5.1 has no such package. There is a
>> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
>> thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>>
>> Was this intentional?
>>
>> More importantly, is there some rough specification for what packages we
>> should be able to expect in this S3 bucket with every release?
>>
>> This is important for those of us who depend on this publishing venue
>> (e.g. spark-ec2 and related tools).
>>
>> Nick
>>
>>
>
>
Re: Spark 1.5.1 - Scala 2.10 - Hadoop 1 package is missing from S3
Posted by Ted Yu <yu...@gmail.com>.
hadoop1 package for Scala 2.10 wasn't in RC1 either:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> I’m looking here:
>
> https://s3.amazonaws.com/spark-related-packages/
>
> I believe this is where one set of official packages is published. Please
> correct me if this is not the case.
>
> It appears that almost every version of Spark up to and including 1.5.0
> has included a --bin-hadoop1.tgz release (e.g. spark-1.5.0-bin-hadoop1.tgz
> ).
>
> However, 1.5.1 has no such package. There is a
> spark-1.5.1-bin-hadoop1-scala2.11.tgz package, but this is a separate
> thing. (1.5.0 also has a hadoop1-scala2.11 package.)
>
> Was this intentional?
>
> More importantly, is there some rough specification for what packages we
> should be able to expect in this S3 bucket with every release?
>
> This is important for those of us who depend on this publishing venue
> (e.g. spark-ec2 and related tools).
>
> Nick
>
>