You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "MrAsanjar ." <af...@gmail.com> on 2017/04/11 16:53:48 UTC
spark 1.6.x package in bigtop 1.2 repo
Hi team,
Could anyone explain why we have spark-extras , a spark 1.6 package,
added to the ubuntu-16.04 repository for Bigtop 1.2? It should be all
spark 2.1 packages and not 1.6.
List of spark packages in Bigtop 1.2 repo:
spark-core/stable,now 2.1.0-1 all [installed]
spark-datanucleus/stable 2.1.0-1 all
spark-external/stable 2.1.0-1 all
spark-extras/stable 1.6.2-1 all <<<<< it should not be there, correct?
spark-history-server/stable,now 2.1.0-1 all [installed]
spark-master/stable,now 2.1.0-1 all [installed]
spark-python/stable 2.1.0-1 all
spark-thriftserver/stable 2.1.0-1 all
spark-worker/stable,now 2.1.0-1 all [installed]
spark-yarn-shuffle/stable 2.1.0-1 all
Re: spark 1.6.x package in bigtop 1.2 repo
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
so... we are good on this one?
Thanks,
Roman.
On Tue, Apr 11, 2017 at 12:56 PM, MrAsanjar . <af...@gmail.com> wrote:
> sounds reasonable, thanks
>
> On Tue, Apr 11, 2017 at 2:07 PM, Evans Ye <ev...@apache.org> wrote:
>
>> So the reason for that is you already have spark 1.6 installed and tried to
>> upgrade to 2.1 by apt-get update.
>>
>> You can see that spark-extras has been renamed to spark-externals:
>>
>> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.
>> 0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
>>
>> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.
>> 0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
>>
>> I'm not quite sure what's inside that package. But it seems to be out of
>> dependency.
>>
>> In general, your path of upgrade is important and worth to be document.
>> I think we can have an Upgrade guide for these issues.
>>
>>
>> 2017-04-12 2:55 GMT+08:00 MrAsanjar . <af...@gmail.com>:
>>
>> > How to reproduce:
>> > 1) Add Bigtop 1.2 repository
>> > wget -O-
>> > http://archive.apache.org/dist/bigtop/bigtop-1.2.0/repos/GPG-KEY-bigtop
>> |
>> > sudo apt-key add -
>> > wget -O /etc/apt/sources.list.d/bigtop-1.2.0.list
>> > https://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/
>> > ubuntu16.04/bigtop.list
>> > sudo apt-get -y update
>> > 2) Query for spark packages by executing command "apt list | grep spark"
>> > result:
>> > spark-core/stable,now 2.1.0-1 all [installed]
>> > spark-datanucleus/stable 2.1.0-1 all
>> > spark-external/stable 2.1.0-1 all
>> > spark-extras/stable 1.6.2-1 all
>> > spark-history-server/stable,now 2.1.0-1 all [installed]
>> > spark-master/stable,now 2.1.0-1 all [installed]
>> > spark-python/stable 2.1.0-1 all
>> > spark-thriftserver/stable 2.1.0-1 all
>> > spark-worker/stable,now 2.1.0-1 all [installed]
>> > spark-yarn-shuffle/stable 2.1.0-1 all
>> >
>> >
>> >
>> >
>> > On Tue, Apr 11, 2017 at 1:29 PM, Evans Ye <ev...@apache.org> wrote:
>> >
>> > > We still packages spark1 to workaround some components compatibility
>> > issue.
>> > > I guess that's why you got 1.6.2 available from 1.2 repo.
>> > >
>> > > How you get these packages installed?
>> > >
>> > > 2017-04-12 0:53 GMT+08:00 MrAsanjar . <af...@gmail.com>:
>> > >
>> > > > Hi team,
>> > > > Could anyone explain why we have spark-extras , a spark 1.6 package,
>> > > > added to the ubuntu-16.04 repository for Bigtop 1.2? It should be
>> all
>> > > > spark 2.1 packages and not 1.6.
>> > > >
>> > > > List of spark packages in Bigtop 1.2 repo:
>> > > >
>> > > > spark-core/stable,now 2.1.0-1 all [installed]
>> > > > spark-datanucleus/stable 2.1.0-1 all
>> > > > spark-external/stable 2.1.0-1 all
>> > > > spark-extras/stable 1.6.2-1 all <<<<< it should not be there,
>> correct?
>> > > > spark-history-server/stable,now 2.1.0-1 all [installed]
>> > > > spark-master/stable,now 2.1.0-1 all [installed]
>> > > > spark-python/stable 2.1.0-1 all
>> > > > spark-thriftserver/stable 2.1.0-1 all
>> > > > spark-worker/stable,now 2.1.0-1 all [installed]
>> > > > spark-yarn-shuffle/stable 2.1.0-1 all
>> > > >
>> > >
>> >
>>
Re: spark 1.6.x package in bigtop 1.2 repo
Posted by "MrAsanjar ." <af...@gmail.com>.
sounds reasonable, thanks
On Tue, Apr 11, 2017 at 2:07 PM, Evans Ye <ev...@apache.org> wrote:
> So the reason for that is you already have spark 1.6 installed and tried to
> upgrade to 2.1 by apt-get update.
>
> You can see that spark-extras has been renamed to spark-externals:
>
> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.
> 0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
>
> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.
> 0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
>
> I'm not quite sure what's inside that package. But it seems to be out of
> dependency.
>
> In general, your path of upgrade is important and worth to be document.
> I think we can have an Upgrade guide for these issues.
>
>
> 2017-04-12 2:55 GMT+08:00 MrAsanjar . <af...@gmail.com>:
>
> > How to reproduce:
> > 1) Add Bigtop 1.2 repository
> > wget -O-
> > http://archive.apache.org/dist/bigtop/bigtop-1.2.0/repos/GPG-KEY-bigtop
> |
> > sudo apt-key add -
> > wget -O /etc/apt/sources.list.d/bigtop-1.2.0.list
> > https://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/
> > ubuntu16.04/bigtop.list
> > sudo apt-get -y update
> > 2) Query for spark packages by executing command "apt list | grep spark"
> > result:
> > spark-core/stable,now 2.1.0-1 all [installed]
> > spark-datanucleus/stable 2.1.0-1 all
> > spark-external/stable 2.1.0-1 all
> > spark-extras/stable 1.6.2-1 all
> > spark-history-server/stable,now 2.1.0-1 all [installed]
> > spark-master/stable,now 2.1.0-1 all [installed]
> > spark-python/stable 2.1.0-1 all
> > spark-thriftserver/stable 2.1.0-1 all
> > spark-worker/stable,now 2.1.0-1 all [installed]
> > spark-yarn-shuffle/stable 2.1.0-1 all
> >
> >
> >
> >
> > On Tue, Apr 11, 2017 at 1:29 PM, Evans Ye <ev...@apache.org> wrote:
> >
> > > We still packages spark1 to workaround some components compatibility
> > issue.
> > > I guess that's why you got 1.6.2 available from 1.2 repo.
> > >
> > > How you get these packages installed?
> > >
> > > 2017-04-12 0:53 GMT+08:00 MrAsanjar . <af...@gmail.com>:
> > >
> > > > Hi team,
> > > > Could anyone explain why we have spark-extras , a spark 1.6 package,
> > > > added to the ubuntu-16.04 repository for Bigtop 1.2? It should be
> all
> > > > spark 2.1 packages and not 1.6.
> > > >
> > > > List of spark packages in Bigtop 1.2 repo:
> > > >
> > > > spark-core/stable,now 2.1.0-1 all [installed]
> > > > spark-datanucleus/stable 2.1.0-1 all
> > > > spark-external/stable 2.1.0-1 all
> > > > spark-extras/stable 1.6.2-1 all <<<<< it should not be there,
> correct?
> > > > spark-history-server/stable,now 2.1.0-1 all [installed]
> > > > spark-master/stable,now 2.1.0-1 all [installed]
> > > > spark-python/stable 2.1.0-1 all
> > > > spark-thriftserver/stable 2.1.0-1 all
> > > > spark-worker/stable,now 2.1.0-1 all [installed]
> > > > spark-yarn-shuffle/stable 2.1.0-1 all
> > > >
> > >
> >
>
Re: spark 1.6.x package in bigtop 1.2 repo
Posted by Evans Ye <ev...@apache.org>.
So the reason for that is you already have spark 1.6 installed and tried to
upgrade to 2.1 by apt-get update.
You can see that spark-extras has been renamed to spark-externals:
https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.2.0/OS=ubuntu-16.04/lastSuccessfulBuild/artifact/output/spark/
I'm not quite sure what's inside that package. But it seems to be out of
dependency.
In general, your path of upgrade is important and worth to be document.
I think we can have an Upgrade guide for these issues.
2017-04-12 2:55 GMT+08:00 MrAsanjar . <af...@gmail.com>:
> How to reproduce:
> 1) Add Bigtop 1.2 repository
> wget -O-
> http://archive.apache.org/dist/bigtop/bigtop-1.2.0/repos/GPG-KEY-bigtop |
> sudo apt-key add -
> wget -O /etc/apt/sources.list.d/bigtop-1.2.0.list
> https://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/
> ubuntu16.04/bigtop.list
> sudo apt-get -y update
> 2) Query for spark packages by executing command "apt list | grep spark"
> result:
> spark-core/stable,now 2.1.0-1 all [installed]
> spark-datanucleus/stable 2.1.0-1 all
> spark-external/stable 2.1.0-1 all
> spark-extras/stable 1.6.2-1 all
> spark-history-server/stable,now 2.1.0-1 all [installed]
> spark-master/stable,now 2.1.0-1 all [installed]
> spark-python/stable 2.1.0-1 all
> spark-thriftserver/stable 2.1.0-1 all
> spark-worker/stable,now 2.1.0-1 all [installed]
> spark-yarn-shuffle/stable 2.1.0-1 all
>
>
>
>
> On Tue, Apr 11, 2017 at 1:29 PM, Evans Ye <ev...@apache.org> wrote:
>
> > We still packages spark1 to workaround some components compatibility
> issue.
> > I guess that's why you got 1.6.2 available from 1.2 repo.
> >
> > How you get these packages installed?
> >
> > 2017-04-12 0:53 GMT+08:00 MrAsanjar . <af...@gmail.com>:
> >
> > > Hi team,
> > > Could anyone explain why we have spark-extras , a spark 1.6 package,
> > > added to the ubuntu-16.04 repository for Bigtop 1.2? It should be all
> > > spark 2.1 packages and not 1.6.
> > >
> > > List of spark packages in Bigtop 1.2 repo:
> > >
> > > spark-core/stable,now 2.1.0-1 all [installed]
> > > spark-datanucleus/stable 2.1.0-1 all
> > > spark-external/stable 2.1.0-1 all
> > > spark-extras/stable 1.6.2-1 all <<<<< it should not be there, correct?
> > > spark-history-server/stable,now 2.1.0-1 all [installed]
> > > spark-master/stable,now 2.1.0-1 all [installed]
> > > spark-python/stable 2.1.0-1 all
> > > spark-thriftserver/stable 2.1.0-1 all
> > > spark-worker/stable,now 2.1.0-1 all [installed]
> > > spark-yarn-shuffle/stable 2.1.0-1 all
> > >
> >
>
Re: spark 1.6.x package in bigtop 1.2 repo
Posted by "MrAsanjar ." <af...@gmail.com>.
How to reproduce:
1) Add Bigtop 1.2 repository
wget -O-
http://archive.apache.org/dist/bigtop/bigtop-1.2.0/repos/GPG-KEY-bigtop |
sudo apt-key add -
wget -O /etc/apt/sources.list.d/bigtop-1.2.0.list
https://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/ubuntu16.04/bigtop.list
sudo apt-get -y update
2) Query for spark packages by executing command "apt list | grep spark"
result:
spark-core/stable,now 2.1.0-1 all [installed]
spark-datanucleus/stable 2.1.0-1 all
spark-external/stable 2.1.0-1 all
spark-extras/stable 1.6.2-1 all
spark-history-server/stable,now 2.1.0-1 all [installed]
spark-master/stable,now 2.1.0-1 all [installed]
spark-python/stable 2.1.0-1 all
spark-thriftserver/stable 2.1.0-1 all
spark-worker/stable,now 2.1.0-1 all [installed]
spark-yarn-shuffle/stable 2.1.0-1 all
On Tue, Apr 11, 2017 at 1:29 PM, Evans Ye <ev...@apache.org> wrote:
> We still packages spark1 to workaround some components compatibility issue.
> I guess that's why you got 1.6.2 available from 1.2 repo.
>
> How you get these packages installed?
>
> 2017-04-12 0:53 GMT+08:00 MrAsanjar . <af...@gmail.com>:
>
> > Hi team,
> > Could anyone explain why we have spark-extras , a spark 1.6 package,
> > added to the ubuntu-16.04 repository for Bigtop 1.2? It should be all
> > spark 2.1 packages and not 1.6.
> >
> > List of spark packages in Bigtop 1.2 repo:
> >
> > spark-core/stable,now 2.1.0-1 all [installed]
> > spark-datanucleus/stable 2.1.0-1 all
> > spark-external/stable 2.1.0-1 all
> > spark-extras/stable 1.6.2-1 all <<<<< it should not be there, correct?
> > spark-history-server/stable,now 2.1.0-1 all [installed]
> > spark-master/stable,now 2.1.0-1 all [installed]
> > spark-python/stable 2.1.0-1 all
> > spark-thriftserver/stable 2.1.0-1 all
> > spark-worker/stable,now 2.1.0-1 all [installed]
> > spark-yarn-shuffle/stable 2.1.0-1 all
> >
>
Re: spark 1.6.x package in bigtop 1.2 repo
Posted by Evans Ye <ev...@apache.org>.
We still packages spark1 to workaround some components compatibility issue.
I guess that's why you got 1.6.2 available from 1.2 repo.
How you get these packages installed?
2017-04-12 0:53 GMT+08:00 MrAsanjar . <af...@gmail.com>:
> Hi team,
> Could anyone explain why we have spark-extras , a spark 1.6 package,
> added to the ubuntu-16.04 repository for Bigtop 1.2? It should be all
> spark 2.1 packages and not 1.6.
>
> List of spark packages in Bigtop 1.2 repo:
>
> spark-core/stable,now 2.1.0-1 all [installed]
> spark-datanucleus/stable 2.1.0-1 all
> spark-external/stable 2.1.0-1 all
> spark-extras/stable 1.6.2-1 all <<<<< it should not be there, correct?
> spark-history-server/stable,now 2.1.0-1 all [installed]
> spark-master/stable,now 2.1.0-1 all [installed]
> spark-python/stable 2.1.0-1 all
> spark-thriftserver/stable 2.1.0-1 all
> spark-worker/stable,now 2.1.0-1 all [installed]
> spark-yarn-shuffle/stable 2.1.0-1 all
>