You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Roman Shaposhnik <ro...@shaposhnik.org> on 2017/03/24 00:42:21 UTC
Releasing 1.2 for non-x86 architectures
Hi!
recent issues surrounding builds on two of
our two non-x86 platforms got me thinking
that while it is fun to build on them (to iron out
kinds in JDK and what not) we don't actually
have to release the packages for most of the
projects since all those are arch independent.
The only ones that ARE arch dependent would be:
bigtop-jsvc
gpdb
hadoop
hue
qfs
tajo
zookeeper
The question then becomes how do you structure
the repos. Do you have on repo for arch independent
stuff and one for each arch? Can you have all the
packages in a single repo?
What are the good practices around this?
I'm especially looking for advice from Canonical guys
and Linaro folks.
Thanks,
Roman.
Re: Releasing 1.2 for non-x86 architectures
Posted by Ganesh Raju <ga...@linaro.org>.
Roman,
We will run on ARM as well and let you know.
Ganesh
On Mon, Mar 27, 2017 at 11:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Mon, Mar 27, 2017 at 5:49 PM, MrAsanjar . <af...@gmail.com> wrote:
> > +1 on Roman proposal..
> > I ran multiple small smoke tests with x86 zeppelin, spark 1.6.x and tez
> on
> > ppc64le and they looked fine. Hadoop however failed when native libraries
> > were included, as expected.
> > I would do more extensive testing when repo becomes available..
>
> Great! Thanks for the testing. Btw, the following now has the non-x86 repos
> laid out according to the proposal:
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-repos/
>
> It would be great if ARM folks can test as well.
>
> Thanks,
> Roman.
>
--
IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Releasing 1.2 for non-x86 architectures
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Mar 27, 2017 at 5:49 PM, MrAsanjar . <af...@gmail.com> wrote:
> +1 on Roman proposal..
> I ran multiple small smoke tests with x86 zeppelin, spark 1.6.x and tez on
> ppc64le and they looked fine. Hadoop however failed when native libraries
> were included, as expected.
> I would do more extensive testing when repo becomes available..
Great! Thanks for the testing. Btw, the following now has the non-x86 repos
laid out according to the proposal:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-repos/
It would be great if ARM folks can test as well.
Thanks,
Roman.
Re: Releasing 1.2 for non-x86 architectures
Posted by "MrAsanjar ." <af...@gmail.com>.
+1 on Roman proposal..
I ran multiple small smoke tests with x86 zeppelin, spark 1.6.x and tez on
ppc64le and they looked fine. Hadoop however failed when native libraries
were included, as expected.
I would do more extensive testing when repo becomes available..
On Mon, Mar 27, 2017 at 12:12 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Mon, Mar 27, 2017 at 10:05 AM, MrAsanjar . <af...@gmail.com> wrote:
> > we need to be careful here, some components have build dependencies to
> > other native components. Casing point, Zeppelin, Tez, and Amabri have
> > dependencies to frontend-maven-plugin that pulls in native NodeJS binary
> > based on the build system.
>
> Those still are built-time dependencies. If you know anything that makes
> a difference during the runtime -- please let us know.
>
> Thanks,
> Roman.
>
Re: Releasing 1.2 for non-x86 architectures
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Mar 27, 2017 at 10:05 AM, MrAsanjar . <af...@gmail.com> wrote:
> we need to be careful here, some components have build dependencies to
> other native components. Casing point, Zeppelin, Tez, and Amabri have
> dependencies to frontend-maven-plugin that pulls in native NodeJS binary
> based on the build system.
Those still are built-time dependencies. If you know anything that makes
a difference during the runtime -- please let us know.
Thanks,
Roman.
Re: Releasing 1.2 for non-x86 architectures
Posted by "MrAsanjar ." <af...@gmail.com>.
we need to be careful here, some components have build dependencies to
other native components. Casing point, Zeppelin, Tez, and Amabri have
dependencies to frontend-maven-plugin that pulls in native NodeJS binary
based on the build system.
On Sat, Mar 25, 2017 at 2:04 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> Those workaround only matter during build time. Once the package is
> built it is truly
> arch independent. IOW, we can build on x86 and use it anywhere.
>
> This is a bit confusing since node itself is NOT arch independent.
> However, none of
> these packages actually need node -- they all use it to
> preprocess/bundle JS during
> compile time.
>
> Makes sense?
>
> Thanks,
> Roman.
>
> On Sat, Mar 25, 2017 at 6:26 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> > Hi Roman,
> >
> > BTW: There are _arch dependent_ package workarounds in _arch independent_
> > packages ! I think we should mark the package as arch dependent in order
> to
> > make sure.
> >
> > The one which comes to mind is tez:
> >
> > See src/common/tez/do-component-build ;-(
> >
> > Will file a JIRA.
> >
> > Olaf
> >
> >
> > Am 25.03.2017 um 13:57 schrieb Olaf Flebbe <of...@oflebbe.de>:
> >
> > Hi,
> >
> > On the debian side I was assuming that reprepro will handle this for you.
> > Simply add different arch specific packages (which can be found by name).
> >
> > I am not sure about the yum side of things.
> >
> >
> > Olaf
> >
> >
> > Am 24.03.2017 um 01:42 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
> >
> > Hi!
> >
> > recent issues surrounding builds on two of
> > our two non-x86 platforms got me thinking
> > that while it is fun to build on them (to iron out
> > kinds in JDK and what not) we don't actually
> > have to release the packages for most of the
> > projects since all those are arch independent.
> > The only ones that ARE arch dependent would be:
> > bigtop-jsvc
> > gpdb
> > hadoop
> > hue
> > qfs
> > tajo
> > zookeeper
> >
> > The question then becomes how do you structure
> > the repos. Do you have on repo for arch independent
> > stuff and one for each arch? Can you have all the
> > packages in a single repo?
> >
> > What are the good practices around this?
> >
> > I'm especially looking for advice from Canonical guys
> > and Linaro folks.
> >
> > Thanks,
> > Roman.
> >
> >
> >
>
Re: Releasing 1.2 for non-x86 architectures
Posted by Roman Shaposhnik <rv...@apache.org>.
Those workaround only matter during build time. Once the package is
built it is truly
arch independent. IOW, we can build on x86 and use it anywhere.
This is a bit confusing since node itself is NOT arch independent.
However, none of
these packages actually need node -- they all use it to
preprocess/bundle JS during
compile time.
Makes sense?
Thanks,
Roman.
On Sat, Mar 25, 2017 at 6:26 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi Roman,
>
> BTW: There are _arch dependent_ package workarounds in _arch independent_
> packages ! I think we should mark the package as arch dependent in order to
> make sure.
>
> The one which comes to mind is tez:
>
> See src/common/tez/do-component-build ;-(
>
> Will file a JIRA.
>
> Olaf
>
>
> Am 25.03.2017 um 13:57 schrieb Olaf Flebbe <of...@oflebbe.de>:
>
> Hi,
>
> On the debian side I was assuming that reprepro will handle this for you.
> Simply add different arch specific packages (which can be found by name).
>
> I am not sure about the yum side of things.
>
>
> Olaf
>
>
> Am 24.03.2017 um 01:42 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>
> Hi!
>
> recent issues surrounding builds on two of
> our two non-x86 platforms got me thinking
> that while it is fun to build on them (to iron out
> kinds in JDK and what not) we don't actually
> have to release the packages for most of the
> projects since all those are arch independent.
> The only ones that ARE arch dependent would be:
> bigtop-jsvc
> gpdb
> hadoop
> hue
> qfs
> tajo
> zookeeper
>
> The question then becomes how do you structure
> the repos. Do you have on repo for arch independent
> stuff and one for each arch? Can you have all the
> packages in a single repo?
>
> What are the good practices around this?
>
> I'm especially looking for advice from Canonical guys
> and Linaro folks.
>
> Thanks,
> Roman.
>
>
>
Re: Releasing 1.2 for non-x86 architectures
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Roman,
BTW: There are _arch dependent_ package workarounds in _arch independent_ packages ! I think we should mark the package as arch dependent in order to make sure.
The one which comes to mind is tez:
See src/common/tez/do-component-build ;-(
Will file a JIRA.
Olaf
> Am 25.03.2017 um 13:57 schrieb Olaf Flebbe <of...@oflebbe.de>:
>
> Hi,
>
> On the debian side I was assuming that reprepro will handle this for you. Simply add different arch specific packages (which can be found by name).
>
> I am not sure about the yum side of things.
>
>
> Olaf
>
>
>> Am 24.03.2017 um 01:42 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>>
>> Hi!
>>
>> recent issues surrounding builds on two of
>> our two non-x86 platforms got me thinking
>> that while it is fun to build on them (to iron out
>> kinds in JDK and what not) we don't actually
>> have to release the packages for most of the
>> projects since all those are arch independent.
>> The only ones that ARE arch dependent would be:
>> bigtop-jsvc
>> gpdb
>> hadoop
>> hue
>> qfs
>> tajo
>> zookeeper
>>
>> The question then becomes how do you structure
>> the repos. Do you have on repo for arch independent
>> stuff and one for each arch? Can you have all the
>> packages in a single repo?
>>
>> What are the good practices around this?
>>
>> I'm especially looking for advice from Canonical guys
>> and Linaro folks.
>>
>> Thanks,
>> Roman.
>
Re: Releasing 1.2 for non-x86 architectures
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
On the debian side I was assuming that reprepro will handle this for you. Simply add different arch specific packages (which can be found by name).
I am not sure about the yum side of things.
Olaf
> Am 24.03.2017 um 01:42 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>
> Hi!
>
> recent issues surrounding builds on two of
> our two non-x86 platforms got me thinking
> that while it is fun to build on them (to iron out
> kinds in JDK and what not) we don't actually
> have to release the packages for most of the
> projects since all those are arch independent.
> The only ones that ARE arch dependent would be:
> bigtop-jsvc
> gpdb
> hadoop
> hue
> qfs
> tajo
> zookeeper
>
> The question then becomes how do you structure
> the repos. Do you have on repo for arch independent
> stuff and one for each arch? Can you have all the
> packages in a single repo?
>
> What are the good practices around this?
>
> I'm especially looking for advice from Canonical guys
> and Linaro folks.
>
> Thanks,
> Roman.