You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2014/08/26 22:16:02 UTC
spark-ec2 1.0.2 creates EC2 cluster at wrong version
I downloaded the source code release for 1.0.2 from here
<http://spark.apache.org/downloads.html> and launched an EC2 cluster using
spark-ec2.
After the cluster finishes launching, I fire up the shell and check the
version:
scala> sc.version
res1: String = 1.0.1
The startup banner also shows the same thing. Hmm...
So I dig around and find that the spark_ec2.py script has the default Spark
version set to 1.0.1.
Derp.
parser.add_option("-v", "--spark-version", default="1.0.1",
help="Version of Spark to use: 'X.Y.Z' or a specific git hash")
Is there any way to fix the release? It’s a minor issue, but could be very
confusing. And how can we prevent this from happening again?
Nick
Re: spark-ec2 1.0.2 creates EC2 cluster at wrong version
Posted by Tathagata Das <ta...@gmail.com>.
Yes, this was an oversight on my part. I have opened a JIRA for this.
https://issues.apache.org/jira/browse/SPARK-3242
For the time being the workaround should be providing the version 1.0.2
explicitly as part of the script.
TD
On Tue, Aug 26, 2014 at 6:39 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> This shouldn't be a chicken-and-egg problem, since the script fetches the
> AMI from a known URL. Seems like an issue in publishing this release.
>
> On August 26, 2014 at 1:24:45 PM, Shivaram Venkataraman (
> shivaram@eecs.berkeley.edu) wrote:
>
> This is a chicken and egg problem in some sense. We can't change the ec2
> script till we have made the release and uploaded the binaries -- But once
> that is done, we can't update the script.
>
> I think the model we support so far is that you can launch the latest
> spark version from the master branch on github. I guess we can try to add
> something in the release process that updates the script but doesn't commit
> it ? The release managers might be able to add more.
>
> Thanks
> Shivaram
>
>
> On Tue, Aug 26, 2014 at 1:16 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
> > I downloaded the source code release for 1.0.2 from here
> > <http://spark.apache.org/downloads.html> and launched an EC2 cluster
> using
> > spark-ec2.
> >
> > After the cluster finishes launching, I fire up the shell and check the
> > version:
> >
> > scala> sc.version
> > res1: String = 1.0.1
> >
> > The startup banner also shows the same thing. Hmm...
> >
> > So I dig around and find that the spark_ec2.py script has the default
> Spark
> > version set to 1.0.1.
> >
> > Derp.
> >
> > parser.add_option("-v", "--spark-version", default="1.0.1",
> > help="Version of Spark to use: 'X.Y.Z' or a specific git hash")
> >
> > Is there any way to fix the release? It’s a minor issue, but could be
> very
> > confusing. And how can we prevent this from happening again?
> >
> > Nick
> >
> >
>
Re: spark-ec2 1.0.2 creates EC2 cluster at wrong version
Posted by Matei Zaharia <ma...@gmail.com>.
This shouldn't be a chicken-and-egg problem, since the script fetches the AMI from a known URL. Seems like an issue in publishing this release.
On August 26, 2014 at 1:24:45 PM, Shivaram Venkataraman (shivaram@eecs.berkeley.edu) wrote:
This is a chicken and egg problem in some sense. We can't change the ec2
script till we have made the release and uploaded the binaries -- But once
that is done, we can't update the script.
I think the model we support so far is that you can launch the latest
spark version from the master branch on github. I guess we can try to add
something in the release process that updates the script but doesn't commit
it ? The release managers might be able to add more.
Thanks
Shivaram
On Tue, Aug 26, 2014 at 1:16 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I downloaded the source code release for 1.0.2 from here
> <http://spark.apache.org/downloads.html> and launched an EC2 cluster using
> spark-ec2.
>
> After the cluster finishes launching, I fire up the shell and check the
> version:
>
> scala> sc.version
> res1: String = 1.0.1
>
> The startup banner also shows the same thing. Hmm...
>
> So I dig around and find that the spark_ec2.py script has the default Spark
> version set to 1.0.1.
>
> Derp.
>
> parser.add_option("-v", "--spark-version", default="1.0.1",
> help="Version of Spark to use: 'X.Y.Z' or a specific git hash")
>
> Is there any way to fix the release? It’s a minor issue, but could be very
> confusing. And how can we prevent this from happening again?
>
> Nick
>
>
Re: spark-ec2 1.0.2 creates EC2 cluster at wrong version
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
This is a chicken and egg problem in some sense. We can't change the ec2
script till we have made the release and uploaded the binaries -- But once
that is done, we can't update the script.
I think the model we support so far is that you can launch the latest
spark version from the master branch on github. I guess we can try to add
something in the release process that updates the script but doesn't commit
it ? The release managers might be able to add more.
Thanks
Shivaram
On Tue, Aug 26, 2014 at 1:16 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I downloaded the source code release for 1.0.2 from here
> <http://spark.apache.org/downloads.html> and launched an EC2 cluster using
> spark-ec2.
>
> After the cluster finishes launching, I fire up the shell and check the
> version:
>
> scala> sc.version
> res1: String = 1.0.1
>
> The startup banner also shows the same thing. Hmm...
>
> So I dig around and find that the spark_ec2.py script has the default Spark
> version set to 1.0.1.
>
> Derp.
>
> parser.add_option("-v", "--spark-version", default="1.0.1",
> help="Version of Spark to use: 'X.Y.Z' or a specific git hash")
>
> Is there any way to fix the release? It’s a minor issue, but could be very
> confusing. And how can we prevent this from happening again?
>
> Nick
>
>