You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Konstantin Boudnik <co...@apache.org> on 2016/02/09 15:06:23 UTC
Build environment
Gents,
have you considered creating and checking-in a wrapper script (run-build.sh or
whatever) for the build, instead of writing lengthy shell-scripts in Jenkins?
Then instead of
docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest /bin/sh -c
"date; \
cd /data....
[another 23 lines of shell script one has to type each time.....]
one would need to run
docker run --rm=true -v `pwd`:/data -w /data -u root rlei/mydocker:latest run-build.sh
and be done with it.
Cos
Re: Build environment
Posted by "Ting(Goden) Yao" <ty...@pivotal.io>.
Filed one https://issues.apache.org/jira/browse/HAWQ-415 to summarize Cos,
Caleb and Lei's comments.
On Tue, Feb 16, 2016 at 9:50 AM Ting(Goden) Yao <ty...@pivotal.io> wrote:
> If we are to include pygresql source code, based on Lei's comment , which
> is ASF compatible license , we just need to include them in NOTICE/LICENSE
> File.
> Lei - can you file a JIRA to track this task if you think this is the
> right approach to go?
>
> On Tue, Feb 16, 2016 at 12:26 AM Lei Chang <le...@apache.org> wrote:
>
>> It is used in management tools (in python), so the issue will be found
>> during running time. I think it is better to add a check during running
>> time.
>>
>>
>> Another observation is that the pygresql library licence (
>> http://www.pygresql.org/copyright.html) looks compatible with Apache
>> licence.
>>
>> And this part is quite error prone during build and install on different
>> platforms (for example on Mac).
>>
>> Any thoughts if we include the source code of this library in hawq? Or do
>> we have any restrictions to include source code with compatible
>> licence from Apache release side? It will save a lot of build and install
>> efforts if it is included.
>>
>> Cheers
>> Lei
>>
>>
>>
>> On Tue, Feb 16, 2016 at 3:47 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>
>> > The main oddity is that I can run build successfully without any of
>> this.
>> >
>> > Cos
>> >
>> > On Fri, Feb 12, 2016 at 04:35PM, Caleb Welton wrote:
>> > > >
>> > > > One other point (which seems a bit weird). What's the point of
>> > > > yum install -y postgresql-devel
>> > > > followed by
>> > > > yum erase -y postgresql postgresql-libs postgresql-devel
>> > >
>> > >
>> > > This is a hacky little oddity that is a result of depending on the
>> Python
>> > > pygresql module in the hawq admin scripts.
>> > >
>> > > Pygresql itself requires postgres (actually HAWQ itself would be
>> fine) in
>> > > order to get the libpq bindings compiled correctly. Thus for the yum
>> > > install of pygresql to work the easiest way is to first install
>> postgres.
>> > >
>> > > Ideally there would be a better way of handling this.
>> > >
>> > >
>> > > Regards,
>> > > Caleb
>> > >
>> > > On Thu, Feb 11, 2016 at 1:16 AM, Konstantin Boudnik <co...@apache.org>
>> > wrote:
>> > >
>> > > > Looking a bit further into build dependencies it seems that the
>> > environment
>> > > > relies very heavily on some 3rd party, maintained by someone and
>> > somewhere
>> > > > repos and libs. While it is up to the community on how they handle
>> > their
>> > > > builds, adding repos and packages which aren't maintained in an open
>> > > > fashion,
>> > > > won't be helping to attract new contributors. Cases in point
>> > > >
>> > > > - https://bintray.com/wangzw/rpm/rpm
>> > > > - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
>> > > > -
>> > > >
>> >
>> http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
>> > > >
>> > > > This certainly will be an integration blocker for the downstream
>> > projects
>> > > > like
>> > > > https://issues.apache.org/jira/browse/BIGTOP-2323
>> > > >
>> > > > One other point (which seems a bit weird). What's the point of
>> > > > yum install -y postgresql-devel
>> > > > followed by
>> > > > yum erase -y postgresql postgresql-libs postgresql-devel
>> > > >
>> > > > Thanks
>> > > > Cos
>> > > >
>> > > > On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
>> > > > > Gents,
>> > > > >
>> > > > > have you considered creating and checking-in a wrapper script
>> > > > (run-build.sh or
>> > > > > whatever) for the build, instead of writing lengthy shell-scripts
>> in
>> > > > Jenkins?
>> > > > > Then instead of
>> > > > >
>> > > > > docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest
>> > /bin/sh
>> > > > -c
>> > > > > "date; \
>> > > > >
>> > > > > cd /data....
>> > > > >
>> > > > > [another 23 lines of shell script one has to type each time.....]
>> > > > >
>> > > > > one would need to run
>> > > > >
>> > > > > docker run --rm=true -v `pwd`:/data -w /data -u root
>> > > > rlei/mydocker:latest run-build.sh
>> > > > >
>> > > > > and be done with it.
>> > > > >
>> > > > > Cos
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> >
>>
>
Re: Build environment
Posted by "Ting(Goden) Yao" <ty...@pivotal.io>.
If we are to include pygresql source code, based on Lei's comment , which
is ASF compatible license , we just need to include them in NOTICE/LICENSE
File.
Lei - can you file a JIRA to track this task if you think this is the right
approach to go?
On Tue, Feb 16, 2016 at 12:26 AM Lei Chang <le...@apache.org> wrote:
> It is used in management tools (in python), so the issue will be found
> during running time. I think it is better to add a check during running
> time.
>
>
> Another observation is that the pygresql library licence (
> http://www.pygresql.org/copyright.html) looks compatible with Apache
> licence.
>
> And this part is quite error prone during build and install on different
> platforms (for example on Mac).
>
> Any thoughts if we include the source code of this library in hawq? Or do
> we have any restrictions to include source code with compatible
> licence from Apache release side? It will save a lot of build and install
> efforts if it is included.
>
> Cheers
> Lei
>
>
>
> On Tue, Feb 16, 2016 at 3:47 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
> > The main oddity is that I can run build successfully without any of this.
> >
> > Cos
> >
> > On Fri, Feb 12, 2016 at 04:35PM, Caleb Welton wrote:
> > > >
> > > > One other point (which seems a bit weird). What's the point of
> > > > yum install -y postgresql-devel
> > > > followed by
> > > > yum erase -y postgresql postgresql-libs postgresql-devel
> > >
> > >
> > > This is a hacky little oddity that is a result of depending on the
> Python
> > > pygresql module in the hawq admin scripts.
> > >
> > > Pygresql itself requires postgres (actually HAWQ itself would be fine)
> in
> > > order to get the libpq bindings compiled correctly. Thus for the yum
> > > install of pygresql to work the easiest way is to first install
> postgres.
> > >
> > > Ideally there would be a better way of handling this.
> > >
> > >
> > > Regards,
> > > Caleb
> > >
> > > On Thu, Feb 11, 2016 at 1:16 AM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > >
> > > > Looking a bit further into build dependencies it seems that the
> > environment
> > > > relies very heavily on some 3rd party, maintained by someone and
> > somewhere
> > > > repos and libs. While it is up to the community on how they handle
> > their
> > > > builds, adding repos and packages which aren't maintained in an open
> > > > fashion,
> > > > won't be helping to attract new contributors. Cases in point
> > > >
> > > > - https://bintray.com/wangzw/rpm/rpm
> > > > - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
> > > > -
> > > >
> >
> http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
> > > >
> > > > This certainly will be an integration blocker for the downstream
> > projects
> > > > like
> > > > https://issues.apache.org/jira/browse/BIGTOP-2323
> > > >
> > > > One other point (which seems a bit weird). What's the point of
> > > > yum install -y postgresql-devel
> > > > followed by
> > > > yum erase -y postgresql postgresql-libs postgresql-devel
> > > >
> > > > Thanks
> > > > Cos
> > > >
> > > > On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> > > > > Gents,
> > > > >
> > > > > have you considered creating and checking-in a wrapper script
> > > > (run-build.sh or
> > > > > whatever) for the build, instead of writing lengthy shell-scripts
> in
> > > > Jenkins?
> > > > > Then instead of
> > > > >
> > > > > docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest
> > /bin/sh
> > > > -c
> > > > > "date; \
> > > > >
> > > > > cd /data....
> > > > >
> > > > > [another 23 lines of shell script one has to type each time.....]
> > > > >
> > > > > one would need to run
> > > > >
> > > > > docker run --rm=true -v `pwd`:/data -w /data -u root
> > > > rlei/mydocker:latest run-build.sh
> > > > >
> > > > > and be done with it.
> > > > >
> > > > > Cos
> > > > >
> > > > >
> > > >
> > > >
> > > >
> >
>
Re: Build environment
Posted by Lei Chang <le...@apache.org>.
It is used in management tools (in python), so the issue will be found
during running time. I think it is better to add a check during running
time.
Another observation is that the pygresql library licence (
http://www.pygresql.org/copyright.html) looks compatible with Apache
licence.
And this part is quite error prone during build and install on different
platforms (for example on Mac).
Any thoughts if we include the source code of this library in hawq? Or do
we have any restrictions to include source code with compatible
licence from Apache release side? It will save a lot of build and install
efforts if it is included.
Cheers
Lei
On Tue, Feb 16, 2016 at 3:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> The main oddity is that I can run build successfully without any of this.
>
> Cos
>
> On Fri, Feb 12, 2016 at 04:35PM, Caleb Welton wrote:
> > >
> > > One other point (which seems a bit weird). What's the point of
> > > yum install -y postgresql-devel
> > > followed by
> > > yum erase -y postgresql postgresql-libs postgresql-devel
> >
> >
> > This is a hacky little oddity that is a result of depending on the Python
> > pygresql module in the hawq admin scripts.
> >
> > Pygresql itself requires postgres (actually HAWQ itself would be fine) in
> > order to get the libpq bindings compiled correctly. Thus for the yum
> > install of pygresql to work the easiest way is to first install postgres.
> >
> > Ideally there would be a better way of handling this.
> >
> >
> > Regards,
> > Caleb
> >
> > On Thu, Feb 11, 2016 at 1:16 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Looking a bit further into build dependencies it seems that the
> environment
> > > relies very heavily on some 3rd party, maintained by someone and
> somewhere
> > > repos and libs. While it is up to the community on how they handle
> their
> > > builds, adding repos and packages which aren't maintained in an open
> > > fashion,
> > > won't be helping to attract new contributors. Cases in point
> > >
> > > - https://bintray.com/wangzw/rpm/rpm
> > > - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
> > > -
> > >
> http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
> > >
> > > This certainly will be an integration blocker for the downstream
> projects
> > > like
> > > https://issues.apache.org/jira/browse/BIGTOP-2323
> > >
> > > One other point (which seems a bit weird). What's the point of
> > > yum install -y postgresql-devel
> > > followed by
> > > yum erase -y postgresql postgresql-libs postgresql-devel
> > >
> > > Thanks
> > > Cos
> > >
> > > On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> > > > Gents,
> > > >
> > > > have you considered creating and checking-in a wrapper script
> > > (run-build.sh or
> > > > whatever) for the build, instead of writing lengthy shell-scripts in
> > > Jenkins?
> > > > Then instead of
> > > >
> > > > docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest
> /bin/sh
> > > -c
> > > > "date; \
> > > >
> > > > cd /data....
> > > >
> > > > [another 23 lines of shell script one has to type each time.....]
> > > >
> > > > one would need to run
> > > >
> > > > docker run --rm=true -v `pwd`:/data -w /data -u root
> > > rlei/mydocker:latest run-build.sh
> > > >
> > > > and be done with it.
> > > >
> > > > Cos
> > > >
> > > >
> > >
> > >
> > >
>
Re: Build environment
Posted by Konstantin Boudnik <co...@apache.org>.
The main oddity is that I can run build successfully without any of this.
Cos
On Fri, Feb 12, 2016 at 04:35PM, Caleb Welton wrote:
> >
> > One other point (which seems a bit weird). What's the point of
> > yum install -y postgresql-devel
> > followed by
> > yum erase -y postgresql postgresql-libs postgresql-devel
>
>
> This is a hacky little oddity that is a result of depending on the Python
> pygresql module in the hawq admin scripts.
>
> Pygresql itself requires postgres (actually HAWQ itself would be fine) in
> order to get the libpq bindings compiled correctly. Thus for the yum
> install of pygresql to work the easiest way is to first install postgres.
>
> Ideally there would be a better way of handling this.
>
>
> Regards,
> Caleb
>
> On Thu, Feb 11, 2016 at 1:16 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Looking a bit further into build dependencies it seems that the environment
> > relies very heavily on some 3rd party, maintained by someone and somewhere
> > repos and libs. While it is up to the community on how they handle their
> > builds, adding repos and packages which aren't maintained in an open
> > fashion,
> > won't be helping to attract new contributors. Cases in point
> >
> > - https://bintray.com/wangzw/rpm/rpm
> > - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
> > -
> > http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
> >
> > This certainly will be an integration blocker for the downstream projects
> > like
> > https://issues.apache.org/jira/browse/BIGTOP-2323
> >
> > One other point (which seems a bit weird). What's the point of
> > yum install -y postgresql-devel
> > followed by
> > yum erase -y postgresql postgresql-libs postgresql-devel
> >
> > Thanks
> > Cos
> >
> > On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> > > Gents,
> > >
> > > have you considered creating and checking-in a wrapper script
> > (run-build.sh or
> > > whatever) for the build, instead of writing lengthy shell-scripts in
> > Jenkins?
> > > Then instead of
> > >
> > > docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest /bin/sh
> > -c
> > > "date; \
> > >
> > > cd /data....
> > >
> > > [another 23 lines of shell script one has to type each time.....]
> > >
> > > one would need to run
> > >
> > > docker run --rm=true -v `pwd`:/data -w /data -u root
> > rlei/mydocker:latest run-build.sh
> > >
> > > and be done with it.
> > >
> > > Cos
> > >
> > >
> >
> >
> >
Re: Build environment
Posted by Caleb Welton <cw...@pivotal.io>.
>
> One other point (which seems a bit weird). What's the point of
> yum install -y postgresql-devel
> followed by
> yum erase -y postgresql postgresql-libs postgresql-devel
This is a hacky little oddity that is a result of depending on the Python
pygresql module in the hawq admin scripts.
Pygresql itself requires postgres (actually HAWQ itself would be fine) in
order to get the libpq bindings compiled correctly. Thus for the yum
install of pygresql to work the easiest way is to first install postgres.
Ideally there would be a better way of handling this.
Regards,
Caleb
On Thu, Feb 11, 2016 at 1:16 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Looking a bit further into build dependencies it seems that the environment
> relies very heavily on some 3rd party, maintained by someone and somewhere
> repos and libs. While it is up to the community on how they handle their
> builds, adding repos and packages which aren't maintained in an open
> fashion,
> won't be helping to attract new contributors. Cases in point
>
> - https://bintray.com/wangzw/rpm/rpm
> - http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
> -
> http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
>
> This certainly will be an integration blocker for the downstream projects
> like
> https://issues.apache.org/jira/browse/BIGTOP-2323
>
> One other point (which seems a bit weird). What's the point of
> yum install -y postgresql-devel
> followed by
> yum erase -y postgresql postgresql-libs postgresql-devel
>
> Thanks
> Cos
>
> On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> > Gents,
> >
> > have you considered creating and checking-in a wrapper script
> (run-build.sh or
> > whatever) for the build, instead of writing lengthy shell-scripts in
> Jenkins?
> > Then instead of
> >
> > docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest /bin/sh
> -c
> > "date; \
> >
> > cd /data....
> >
> > [another 23 lines of shell script one has to type each time.....]
> >
> > one would need to run
> >
> > docker run --rm=true -v `pwd`:/data -w /data -u root
> rlei/mydocker:latest run-build.sh
> >
> > and be done with it.
> >
> > Cos
> >
> >
>
>
>
Re: Build environment
Posted by Konstantin Boudnik <co...@apache.org>.
Looking a bit further into build dependencies it seems that the environment
relies very heavily on some 3rd party, maintained by someone and somewhere
repos and libs. While it is up to the community on how they handle their
builds, adding repos and packages which aren't maintained in an open fashion,
won't be helping to attract new contributors. Cases in point
- https://bintray.com/wangzw/rpm/rpm
- http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
- http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
This certainly will be an integration blocker for the downstream projects like
https://issues.apache.org/jira/browse/BIGTOP-2323
One other point (which seems a bit weird). What's the point of
yum install -y postgresql-devel
followed by
yum erase -y postgresql postgresql-libs postgresql-devel
Thanks
Cos
On Tue, Feb 09, 2016 at 06:06AM, Konstantin Boudnik wrote:
> Gents,
>
> have you considered creating and checking-in a wrapper script (run-build.sh or
> whatever) for the build, instead of writing lengthy shell-scripts in Jenkins?
> Then instead of
>
> docker run --rm=true -v `pwd`:/data -u root rlei/mydocker:latest /bin/sh -c
> "date; \
>
> cd /data....
>
> [another 23 lines of shell script one has to type each time.....]
>
> one would need to run
>
> docker run --rm=true -v `pwd`:/data -w /data -u root rlei/mydocker:latest run-build.sh
>
> and be done with it.
>
> Cos
>
>