You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2020/05/28 18:04:33 UTC

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15735

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................

IMPALA-9107 (part 2): Add script to use the m2 archive tarball

This adds a script to find an appropriate m2 archive
tarball, download it, and use it to prepopulate the
~/.m2 directory.

The script uses the JSON interface for Jenkins to search through
the all-build-options-ub1604 builds on jenkins.impala.io to
find one that:
1. Is building the "master" branch
2. Has the m2_archive.tar.gz
Then, it downloads the m2 archive and uses it to populate ~/.m2.
It does not overwrite or remove any files already in ~/.m2.

The build scripts that call populate_m2_directory.py do not
rely on the script succeeding. They will continue even if
the script fails.

This also modifies the build-all-flag-combinations.sh script
to only build the m2 archive if the GENERATE_M2_ARCHIVE
environment variable is true. GENERATE_M2_ARCHIVE=true will
clear out the ~/.m2 directory to build an accurate m2 archive.
Precommit jobs will use GENERATE_M2_ARCHIVE=false, which
will allow them to use the m2 archive to speed up the build.

Testing:
- Ran gerrify-verify-dryrun
- Tested locally

Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
---
M bin/bootstrap_build.sh
M bin/bootstrap_system.sh
M bin/jenkins/build-all-flag-combinations.sh
A bin/jenkins/populate_m2_directory.py
4 files changed, 195 insertions(+), 2 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/15735/5
--
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/build-all-flag-combinations.sh@65
PS6, Line 65: mkdir -p ${TMP_DIR}
> nit: mktemp above will actually create the directory, so this is not necess
Good point, removed


http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/populate_m2_directory.py
File bin/jenkins/populate_m2_directory.py:

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/populate_m2_directory.py@25
PS6, Line 25: ALL_BUILD_OPTIONS_JOB = "all-build-options-ub1604"
> The script would gain some flexibility if these constants were exposed as p
I think I would like for us to add that only if when we need it.



-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jun 2020 20:50:29 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/6287/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jun 2020 21:17:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/6148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2020 18:56:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.

Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 6: Code-Review+2

(2 comments)

LGTM

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/build-all-flag-combinations.sh@65
PS6, Line 65: mkdir -p ${TMP_DIR}
nit: mktemp above will actually create the directory, so this is not necessary (although harmless because of the '-p').


http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/populate_m2_directory.py
File bin/jenkins/populate_m2_directory.py:

http://gerrit.cloudera.org:8080/#/c/15735/6/bin/jenkins/populate_m2_directory.py@25
PS6, Line 25: ALL_BUILD_OPTIONS_JOB = "all-build-options-ub1604"
The script would gain some flexibility if these constants were exposed as parameters with default values; but that's OK to happen in a subsequent commit.



-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jun 2020 16:19:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5982/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jun 2020 20:51:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Hello Laszlo Gaal, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15735

to look at the new patch set (#7).

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................

IMPALA-9107 (part 2): Add script to use the m2 archive tarball

This adds a script to find an appropriate m2 archive
tarball, download it, and use it to prepopulate the
~/.m2 directory.

The script uses the JSON interface for Jenkins to search through
the all-build-options-ub1604 builds on jenkins.impala.io to
find one that:
1. Is building the "master" branch
2. Has the m2_archive.tar.gz
Then, it downloads the m2 archive and uses it to populate ~/.m2.
It does not overwrite or remove any files already in ~/.m2.

The build scripts that call populate_m2_directory.py do not
rely on the script succeeding. They will continue even if
the script fails.

This also modifies the build-all-flag-combinations.sh script
to only build the m2 archive if the GENERATE_M2_ARCHIVE
environment variable is true. GENERATE_M2_ARCHIVE=true will
clear out the ~/.m2 directory to build an accurate m2 archive.
Precommit jobs will use GENERATE_M2_ARCHIVE=false, which
will allow them to use the m2 archive to speed up the build.

Testing:
 - Ran gerrify-verify-dryrun
 - Tested locally

Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
---
M bin/bootstrap_build.sh
M bin/bootstrap_system.sh
M bin/jenkins/build-all-flag-combinations.sh
A bin/jenkins/populate_m2_directory.py
4 files changed, 195 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/15735/7
-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jun 2020 01:52:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................


Patch Set 7: Code-Review+2

Carry +2


-- 
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jun 2020 20:50:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9107 (part 2): Add script to use the m2 archive tarball

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15735 )

Change subject: IMPALA-9107 (part 2): Add script to use the m2 archive tarball
......................................................................

IMPALA-9107 (part 2): Add script to use the m2 archive tarball

This adds a script to find an appropriate m2 archive
tarball, download it, and use it to prepopulate the
~/.m2 directory.

The build scripts that call populate_m2_directory.py do not
rely on the script succeeding. They will continue even if
the script fails.

Testing:
- Ran gerrify-verify-dryrun
- Tested locally

Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Reviewed-on: http://gerrit.cloudera.org:8080/15735
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/bootstrap_build.sh
M bin/bootstrap_system.sh
M bin/jenkins/build-all-flag-combinations.sh
A bin/jenkins/populate_m2_directory.py
4 files changed, 195 insertions(+), 4 deletions(-)

Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/15735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Gerrit-Change-Number: 15735
Gerrit-PatchSet: 8
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>