You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@bigtop.apache.org by Julien Eid <je...@gmail.com> on 2014/07/14 20:49:08 UTC

Review Request 23457: Added ability to build certain Bigtop components against SCM.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/
-----------------------------------------------------------

Review request for bigtop.

Repository: bigtop

Description
-------

I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.

How do I use this?
Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
# Hive
HIVE_NAME=hive
HIVE_RELNOTES_NAME=Apache Hive
HIVE_PKG_NAME=hive
HIVE_BASE_VERSION=0.13.0
HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
HIVE_GIT_BRANCH=master
HIVE_RELEASE_VERSION=1
HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
HIVE_DOWNLOAD_PATH=/hive/archive
HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
HIVE_ARCHIVE=$(HIVE_SITE)
$(eval $(call PACKAGE,hive,HIVE))

So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.

I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.

FAQ:
Why didn't you just do git checkout or svn checkout for the components?
That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.

Why not just set _BASE_VERSION to _GIT_BRANCH?
_GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.

Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.

Diffs
-----

Makefile 5d9e209
bigtop.mk 04c1273
packages.gradle a059ff6

Diff: https://reviews.apache.org/r/23457/diff/

Testing
-------

built hbase with both make and gradle

Thanks,

Julien Eid

Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

Posted by Mark Grover <gr...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/#review47971
-----------------------------------------------------------



Makefile
<https://reviews.apache.org/r/23457/#comment84217>

    We some times get code from places other than github.com/apache. Example is hue which we get from github.com/cloudera, so it would be good to reduce the scope of this parameter to just github.


- Mark Grover


On July 14, 2014, 6:49 p.m., Julien Eid wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23457/
> -----------------------------------------------------------
> 
> (Updated July 14, 2014, 6:49 p.m.)
> 
> 
> Review request for bigtop.
> 
> 
> Repository: bigtop
> 
> 
> Description
> -------
> 
> I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.
> 
> How do I use this?
> Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
> # Hive
> HIVE_NAME=hive
> HIVE_RELNOTES_NAME=Apache Hive
> HIVE_PKG_NAME=hive
> HIVE_BASE_VERSION=0.13.0
> HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
> HIVE_GIT_BRANCH=master
> HIVE_RELEASE_VERSION=1
> HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
> HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
> HIVE_DOWNLOAD_PATH=/hive/archive
> HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
> HIVE_ARCHIVE=$(HIVE_SITE)
> $(eval $(call PACKAGE,hive,HIVE))
> 
> So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.
> 
> I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.
> 
> FAQ:
> Why didn't you just do git checkout or svn checkout for the components?
> That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.
> 
> Why not just set _BASE_VERSION to _GIT_BRANCH?
> _GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.
> 
> Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
> I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.
> 
> 
> Diffs
> -----
> 
>   Makefile 5d9e209 
>   bigtop.mk 04c1273 
>   packages.gradle a059ff6 
> 
> Diff: https://reviews.apache.org/r/23457/diff/
> 
> 
> Testing
> -------
> 
> built hbase with both make and gradle
> 
> 
> Thanks,
> 
> Julien Eid
> 
>

Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

Posted by Mark Grover <gr...@gmail.com>.


> On July 24, 2014, midnight, Mark Grover wrote:
> > Ship It!

https://reviews.apache.org/media/uploaded/files/2014/07/21/a95bc189-d985-4ddb-a460-5dd3a84c8828__BIGTOP-1373-extrafile.patch looks good.


- Mark


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/#review48581
-----------------------------------------------------------


On July 21, 2014, 7:49 p.m., Julien Eid wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23457/
> -----------------------------------------------------------
> 
> (Updated July 21, 2014, 7:49 p.m.)
> 
> 
> Review request for bigtop.
> 
> 
> Repository: bigtop
> 
> 
> Description
> -------
> 
> I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.
> 
> How do I use this?
> Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
> # Hive
> HIVE_NAME=hive
> HIVE_RELNOTES_NAME=Apache Hive
> HIVE_PKG_NAME=hive
> HIVE_BASE_VERSION=0.13.0
> HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
> HIVE_GIT_BRANCH=master
> HIVE_RELEASE_VERSION=1
> HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
> HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
> HIVE_DOWNLOAD_PATH=/hive/archive
> HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
> HIVE_ARCHIVE=$(HIVE_SITE)
> $(eval $(call PACKAGE,hive,HIVE))
> 
> So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.
> 
> I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.
> 
> FAQ:
> Why didn't you just do git checkout or svn checkout for the components?
> That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.
> 
> Why not just set _BASE_VERSION to _GIT_BRANCH?
> _GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.
> 
> Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
> I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.
> 
> 
> Diffs
> -----
> 
>   Makefile 5d9e209 
>   bigtop.mk 04c1273 
>   packages.gradle a059ff6 
> 
> Diff: https://reviews.apache.org/r/23457/diff/
> 
> 
> Testing
> -------
> 
> built hbase with both make and gradle
> 
> 
> File Attachments
> ----------------
> 
> This is the other method
>   https://reviews.apache.org/media/uploaded/files/2014/07/21/a95bc189-d985-4ddb-a460-5dd3a84c8828__BIGTOP-1373-extrafile.patch
> 
> 
> Thanks,
> 
> Julien Eid
> 
>

Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

Posted by Mark Grover <gr...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/#review48581
-----------------------------------------------------------

Ship it!


Ship It!

- Mark Grover


On July 21, 2014, 7:49 p.m., Julien Eid wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23457/
> -----------------------------------------------------------
> 
> (Updated July 21, 2014, 7:49 p.m.)
> 
> 
> Review request for bigtop.
> 
> 
> Repository: bigtop
> 
> 
> Description
> -------
> 
> I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.
> 
> How do I use this?
> Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
> # Hive
> HIVE_NAME=hive
> HIVE_RELNOTES_NAME=Apache Hive
> HIVE_PKG_NAME=hive
> HIVE_BASE_VERSION=0.13.0
> HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
> HIVE_GIT_BRANCH=master
> HIVE_RELEASE_VERSION=1
> HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
> HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
> HIVE_DOWNLOAD_PATH=/hive/archive
> HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
> HIVE_ARCHIVE=$(HIVE_SITE)
> $(eval $(call PACKAGE,hive,HIVE))
> 
> So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.
> 
> I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.
> 
> FAQ:
> Why didn't you just do git checkout or svn checkout for the components?
> That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.
> 
> Why not just set _BASE_VERSION to _GIT_BRANCH?
> _GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.
> 
> Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
> I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.
> 
> 
> Diffs
> -----
> 
>   Makefile 5d9e209 
>   bigtop.mk 04c1273 
>   packages.gradle a059ff6 
> 
> Diff: https://reviews.apache.org/r/23457/diff/
> 
> 
> Testing
> -------
> 
> built hbase with both make and gradle
> 
> 
> File Attachments
> ----------------
> 
> This is the other method
>   https://reviews.apache.org/media/uploaded/files/2014/07/21/a95bc189-d985-4ddb-a460-5dd3a84c8828__BIGTOP-1373-extrafile.patch
> 
> 
> Thanks,
> 
> Julien Eid
> 
>

Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

Posted by Julien Eid <je...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/
-----------------------------------------------------------

(Updated July 21, 2014, 7:49 p.m.)


Review request for bigtop.


Changes
-------

This patch has an extra file called bigtop-github.mk where if you do make hbase-deb-git, it will source from bigtop-github.mk instead of bigtop.mk


Repository: bigtop


Description
-------

I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.

How do I use this?
Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
# Hive
HIVE_NAME=hive
HIVE_RELNOTES_NAME=Apache Hive
HIVE_PKG_NAME=hive
HIVE_BASE_VERSION=0.13.0
HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
HIVE_GIT_BRANCH=master
HIVE_RELEASE_VERSION=1
HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
HIVE_DOWNLOAD_PATH=/hive/archive
HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
HIVE_ARCHIVE=$(HIVE_SITE)
$(eval $(call PACKAGE,hive,HIVE))

So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.

I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.

FAQ:
Why didn't you just do git checkout or svn checkout for the components?
That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.

Why not just set _BASE_VERSION to _GIT_BRANCH?
_GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.

Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.


Diffs
-----

  Makefile 5d9e209 
  bigtop.mk 04c1273 
  packages.gradle a059ff6 

Diff: https://reviews.apache.org/r/23457/diff/


Testing
-------

built hbase with both make and gradle


File Attachments (updated)
----------------

This is the other method
  https://reviews.apache.org/media/uploaded/files/2014/07/21/a95bc189-d985-4ddb-a460-5dd3a84c8828__BIGTOP-1373-extrafile.patch


Thanks,

Julien Eid

Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

Posted by Julien Eid <je...@gmail.com>.

(Updated July 21, 2014, 7:48 p.m.)

Review request for bigtop.

Changes
-------

This is a final patch one method of doing this where I described this in the Description. You would modify bigtop.mk for the specific project you want to pull from SCM.

Repository: bigtop

Description
-------

So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.

I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.

Diffs (updated)
-----

Makefile 5d9e209
bigtop.mk 04c1273
packages.gradle a059ff6

Diff: https://reviews.apache.org/r/23457/diff/

Testing
-------

built hbase with both make and gradle

Thanks,

Julien Eid