You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jim Apple (Code Review)" <ge...@cloudera.org> on 2017/10/12 04:42:39 UTC
[Impala-ASF-CR] Make build scripts more friendly to Ubuntu 16.04
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8262
Change subject: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Make build scripts more friendly to Ubuntu 16.04
This commit bundles two changes.
The first extracts from bootstrap_development.sh the commands to
prepare a system to build-and-test without actually doing so. This
enables custom build commands that might not load the test data, for
instance.
The second changes bootstrap_build.sh to work with Ubuntu 16.04. It
should still work with Ubuntu 14.04, but I don't anticipate that being
part of the Jenkins pre-merge job anymore, so I removed that from the
comment at the top of the file explaining what it does.
Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
---
M bin/bootstrap_build.sh
M bin/bootstrap_development.sh
A bin/bootstrap_system.sh
3 files changed, 227 insertions(+), 174 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/8262/1
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 3:
(2 comments)
Sorry to get to this a bit late.
I find the distinctions between the (now 3) bootstrap_* scripts to be really hard to grok. Since we're multiplying scripts in this commit, it seems like we should clarify those distinctions.
I very much like separating bootstrap_system from bootstrap_development, but I think it doesn't make sense to separate it into two scripts. Rather, I think this would be clearer to use if it looked like:
bootstrap_development [install-packages] [checkout-impala] [configure] [build] [loadtestdata] [runtests]
(Or something like it; point being to have on entry point with a few variants.)
The default actions would be whatever bootstrap_development does today.
There's a mild prototype at https://github.com/philz/incubator-impala/commit/e0e7ad3ce07335709987b1b4088ac66ef312e7af if you're interested. This came up for me while trying to time the different parts of the bootstrap script for some Docker work I was doing. In it, I just broke the existing script into ~5 bash functions.
http://gerrit.cloudera.org:8080/#/c/8262/3/bin/bootstrap_build.sh
File bin/bootstrap_build.sh:
http://gerrit.cloudera.org:8080/#/c/8262/3/bin/bootstrap_build.sh@35
PS3, Line 35: python-dev python-setuptools libffi-dev libkrb5-dev
Consider just adding openjdk-${JDK_VERSION}-jdk (and friends) here? I don't know that one apt-get command is faster than two, but it's nice to have them all together.
http://gerrit.cloudera.org:8080/#/c/8262/3/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:
http://gerrit.cloudera.org:8080/#/c/8262/3/bin/bootstrap_system.sh@20
PS3, Line 20: # This script bootstraps a system for Impala development from almost nothing; it is known
This is very heavily replicated with bootstrap_development; highlighting the differences may make sense.
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Oct 2017 16:54:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Hello Lars Volker, Michael Brown, Zoram Thanga,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8262
to look at the new patch set (#2).
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
This commit bundles two changes.
The first extracts from bootstrap_development.sh the commands to
prepare a system to build-and-test without actually doing so. This
enables custom build commands that might not load the test data, for
instance.
The second changes bootstrap_build.sh to work with Ubuntu 16.04. It
should still work with Ubuntu 14.04, but I don't anticipate that being
part of the Jenkins pre-merge job anymore, so I removed that from the
comment at the top of the file explaining what it does.
Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
---
M bin/bootstrap_build.sh
M bin/bootstrap_development.sh
A bin/bootstrap_system.sh
3 files changed, 227 insertions(+), 174 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/8262/2
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 3:
> Patch Set 3:
>
> > Sorry to get to this a bit late.
>
> No apology required.
>
> I agree with all of your suggestions.
>
> If you want to file a ticket and assign it to me or if you want to send me a patch to review, I'd be amenable.
Filed https://issues.apache.org/jira/browse/IMPALA-6079.
I've left it unassigned. It's not clear to me yet whether some other work will lead me right back to this :)
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Oct 2017 22:38:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 2: Verified+1
Verified in https://jenkins.impala.io/job/gerrit-verify-dryrun/1334/
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2017 20:54:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
This commit bundles two changes.
The first extracts from bootstrap_development.sh the commands to
prepare a system to build-and-test without actually doing so. This
enables custom build commands that might not load the test data, for
instance.
The second changes bootstrap_build.sh to work with Ubuntu 16.04. It
should still work with Ubuntu 14.04, but I don't anticipate that being
part of the Jenkins pre-merge job anymore, so I removed that from the
comment at the top of the file explaining what it does.
Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Reviewed-on: http://gerrit.cloudera.org:8080/8262
Reviewed-by: Michael Brown <mi...@cloudera.com>
Tested-by: Jim Apple <jb...@apache.org>
---
M bin/bootstrap_build.sh
M bin/bootstrap_development.sh
A bin/bootstrap_system.sh
3 files changed, 227 insertions(+), 174 deletions(-)
Approvals:
Michael Brown: Looks good to me, approved
Jim Apple: Verified
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2017 20:49:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8262/1/bin/bootstrap_build.sh
File bin/bootstrap_build.sh:
http://gerrit.cloudera.org:8080/#/c/8262/1/bin/bootstrap_build.sh@23
PS1, Line 23: # 1. At least 8GB of free disk space
> Is this still true? I vaguely recollect running out of space in a vm that h
Yes, check the df output: https://jenkins.impala.io/view/Experimental/job/ubuntu-16.04-build-only/4/consoleFull
You may have been loading data, building static (and building all tests), or other developer activities.
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2017 05:00:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] Make build scripts more friendly to Ubuntu 16.04
Posted by "Zoram Thanga (Code Review)" <ge...@cloudera.org>.
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8262/1/bin/bootstrap_build.sh
File bin/bootstrap_build.sh:
http://gerrit.cloudera.org:8080/#/c/8262/1/bin/bootstrap_build.sh@23
PS1, Line 23: # 1. At least 8GB of free disk space
Is this still true? I vaguely recollect running out of space in a vm that had 32GB of disk space.
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2017 04:48:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 3:
> Sorry to get to this a bit late.
No apology required.
I agree with all of your suggestions.
If you want to file a ticket and assign it to me or if you want to send me a patch to review, I'd be amenable.
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2017 05:22:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/8262 )
Change subject: IMPALA-6045: Make build scripts more friendly to Ubuntu 16.04
......................................................................
Patch Set 2:
Passed GVD: https://jenkins.impala.io/job/gerrit-verify-dryrun/1334/
--
To view, visit http://gerrit.cloudera.org:8080/8262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Gerrit-Change-Number: 8262
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2017 20:27:30 +0000
Gerrit-HasComments: No