You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2026/07/15 06:30:30 UTC

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24583


Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................

WIP: KUDU-3788 an option to use pre-built thirdparty components

WIP:
  * add proper description
  * handle binary-already-in-the-source components properly,
    creating archives of the same pre-built flavor for them as well
  * implement clear separation between pre-built and legacy
    option -- that might be relevant if somebody is running multiple
    concurrent builds on the same node for different Kudu versions
    that require different versions of Kudu's thirdparty components
  * add proper rollback, removing half-baked pre-built archive
    if anything fails while creating the archive
  * brush up the patch
  * add in-line docs where necessary
  * address TODOs

Prefix is set to /opt/kudu/thirdparty when using pre-built binaries.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed

DONT_BUILD

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
7 files changed, 873 insertions(+), 276 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#12).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when it doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JARs and similar content
    (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,022 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/12
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 12:

(6 comments)

Thank you for addressing the comments.

Almost there. Just a few nits.

I am not sure if docker builds are automatically tested by dist-cluster or has to be done explicitly. Would be great if you could verify that everything checks out for docker build. Btw, I am fine with addressing any unforeseen issues there in a separate change.

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@80
PS11, Line 80: cut -d. -f1
> As of PS5, it's only the major version, correct.  That's reflected in the c
I may have highlighted wrong place that caused the confusion. I should have highlighted the comment instead. I meant in the comment above at line #76:
++
Detect the host OS version string. Use only major and minor versions,
++


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@116
PS12, Line 116: $version_str
Should this be in quotes to avoid collapse of newlines to spaces and always end up returning 1?


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@253
PS12, Line 253: install_subdir
This is unused, can be removed here and in callers.


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@381
PS12, Line 381: $tarball
s/$tarball/$tarball_name ?


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@388
PS12, Line 388: if [ -z "$staging_root" ]; then
$staging_root will always be non-empty with '/' lying around even if $component and $TP_STAGING_DIR is empty. Maybe add empty check for both separately if the goal is to report the same?


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@413
PS12, Line 413: $tarball
s/$tarball/$tarball_name ?



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 12:17:07 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6: -Verified

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2799/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 22:52:47 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2804/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 03:30:49 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24583/5/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

PS5: 
> Ran the below command in a newly cloned repo with this change
You'd get the same error before this patch.

To use build-thirdparty.sh, first run download-thirdparty.sh.

Otherwise, the standard and documented way of building Kudu's thirdparty is using the build-if-necessary.sh script, not build-thirdparty.sh.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 08:10:56 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2760/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 18 Jul 2026 00:39:21 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 9:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2802/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 00:55:53 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has removed a vote on this change.

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#13).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when it doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JARs and similar content
    (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,027 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/13
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@30
PS5, Line 30: After that, it's
            : possible to upload the result into a sub-directory of the designated
            : S3 bucket s3://cloudera-thirdparty-libs/prebuilt
Question: To leverage this gain, how do we manage the manual upload of pre-built artifacts to S3, given there would be a good number of archives for even a limited and usual configuration/setup (just from CI) combinatorial space?


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@35
PS5, Line 35: distrute
distribute


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: uninstrumeted
uninstrumented


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: /opt/kudu/
Have you considered 'home' directory approach to avoid any first-time directory creation (/opt/kudu/thirdparty) and permission setting challenges, etc. ?

I am guessing single user CI/dev system is the general use case here and concurrent build by multiple users is not something that needs to be considered here?


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@62
PS5, Line 62: Only CMake/GNUmake combination is now used for building
            : 3rd-party components that rely on CMake to bootstrap build environment
Is there a significant increase in build time if not using ninja, especially for LLVM?

I understand that this is supposed to be one-time penalty, considering artifacts would get built and uploaded to S3 in timely fashion. As a result, subsequent builds would download and use those directly, thereby saving time but just for curiosity.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 08:46:05 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#6).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 970 insertions(+), 347 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.

Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24583/5/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

PS5: 
Ran the below command in a newly cloned repo with this change
% USE_PREBUILT_THIRDPARTY=0 ./build-thirdparty.sh
Errors out with:
kudu/thirdparty/src/cmake-3.25.3/bootstrap: No such file or directory



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 06:33:42 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 7: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2797/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 22:14:39 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6: -Verified

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2796/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 18:52:43 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#15).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when it doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JARs and similar content
    (e.g., interpreted scripts, etc.).

CLEAN_THIRDPARTY

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M docker/Dockerfile
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
10 files changed, 1,016 insertions(+), 394 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/15
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 14:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/build-thirdparty.sh@50
PS13, Line 50: 
> nit: Is this still needed? I don't see it referenced anywhere.
Done


http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh@172
PS14, Line 172:   local plevel="${!plevel_var}"
> nit: `$version` is validated just above (errors if empty), but `$plevel`
Good point.  Done.


http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh@330
PS14, Line 330:   return 0
> This `return 0` (combined with the callers at lines 369 and 410 invoking
Good catch!  Indeed, that's an overlook: it seems I'm giving too much credibility to certain AI tools :)

Thanks a lot, this is now fixed.


http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/vars.sh
File thirdparty/vars.sh:

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/vars.sh@52
PS13, Line 52: chain
> nit: Does it make sense to allow users to customize this? For example in a 
No, it doesn't.  This is a part of the pre-compiled PREFIX that must be the same for all the pre-built artifacts.  If this changed, pre-compiled artifacts aren't going to work for -I/-L flags of the compiler/linker, and pkgconfig would return wrong location of the libraries and headers for 3rd-party components, etc.

If regular (i.e. non-superuser) users do not have permissions for /opt/kudu/thirdparty, the script prompts them to grant the necessary permissions prior to commencing the build.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 21:16:45 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2760/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 18 Jul 2026 00:39:35 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh@163
PS4, Line 163:   # Building a variable in the xxx_NAME form: vars.sh contains corresponding
             :   # versioned strings for all the 3rd-party components.
             :   local version_var="${name}_VERSION"
             :   local version="${!version_var}"
             :   if [ -z "$version" ]; then
             :     echo "ERROR: $version_var isn't defined: check vars.sh" >&2
             :     exit 1
             :   fi
             :   local plevel_var="${name}_PATCHLE
> If the patch version is reflected in the value of the corresponding xxx_VER
After some consideration, I added patch version into the name of the pre-built archive.  So, for cmake built for macOS 26.5.2 it looks like below (CMAKE_VERSION=3.25.3, CMAKE_PATCHLEVEL=2):

  cmake-3.25.3.p2.macos-26.x86_64.clang-17.0.tar.xz

Thanks a lot for the input!



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 00:57:20 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.

Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 16: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 16
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 03 Aug 2026 14:24:48 +0000
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#2).

Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................

WIP: KUDU-3788 an option to use pre-built thirdparty components

WIP:
  * add proper description
  * implement clear separation between pre-built and legacy
    option -- that might be relevant if somebody is running multiple
    concurrent builds on the same node for different Kudu versions
    that require different versions of Kudu's thirdparty components
  * add proper rollback, removing half-baked pre-built archive
    if anything fails while creating the archive
  * brush up the patch
  * add in-line docs where necessary
  * address TODOs

Prefix is set to /opt/kudu/thirdparty when using pre-built binaries.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed

DONT_BUILD

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
8 files changed, 964 insertions(+), 321 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 7:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2797/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 20:51:12 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.

Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 13:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/build-thirdparty.sh@50
PS13, Line 50: KUDU_ROOT
nit: Is this still needed? I don't see it referenced anywhere.


http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/vars.sh
File thirdparty/vars.sh:

http://gerrit.cloudera.org:8080/#/c/24583/13/thirdparty/vars.sh@52
PS13, Line 52: /opt/kudu/thirdparty
nit: Does it make sense to allow users to customize this? For example in a case where there are any file permissions blocking them to write into '/opt'?



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 01:33:34 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 14:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2819/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 04:36:13 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2804/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 05:00:29 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

Build Aborted 

http://jenkins.kudu.apache.org/job/pre_commit/2790/ : ABORTED


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 18:08:17 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6:

Build Aborted 

http://jenkins.kudu.apache.org/job/pre_commit/2799/ : ABORTED


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 23:25:25 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#4).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major and minor version (9.2, 24.04, 26.5, etc.)
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distrute only only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumeted,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
8 files changed, 959 insertions(+), 339 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has removed a vote on this change.

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG@16
PS4, Line 16:   * name of the component (cmake, protobuf, etc.)
            :   * version of the component as in $KUDU_HOME/thirdparty/vars.sh
            :   * OS name/flavor (redhat, ubuntu, macos, etc.)
            :   * OS major and minor version (9.2, 24.04, 26.5, etc.)
            :   * CPU architecture (x86_64, aarch64, etc.)
            :   * build toolchain info:
            :     ** compiler family (gcc, clang, etc.)
            :     ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)
> > Do the identifiers know about transitive dependencies?
I think it's a good idea to use only the major version of the OS version.  At least, since we control what's uploaded into the bucket, we can always build on .0 minor version of a particular major release and upload the result artifacts into the bucket.

Done.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 06:02:45 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2796/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 19:45:53 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has removed a vote on this change.

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 8:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2800/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 23:28:37 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 9: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2802/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 02:45:14 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3:

Build Aborted 

http://jenkins.kudu.apache.org/job/pre_commit/2762/ : ABORTED


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Jul 2026 15:02:33 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2761/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Jul 2026 02:50:20 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 15: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2828/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 03:01:46 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 12:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2809/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 22:58:02 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 13: Verified+1

Unrelated test failures:
  * RELEASE: KUDU-3275
  * TSAN: known flaky Java tests


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 01:27:51 +0000
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................


Patch Set 2: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2757/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jul 2026 03:04:25 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 15: Verified+1

Unrelated failures of known Java flaky tests (org.apache.kudu.client.TestSecurity, etc.) in TSAN.


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 03:18:07 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.

Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 15: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Aug 2026 09:39:36 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2795/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 18:36:35 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2803/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 03:17:16 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2795/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 18:08:16 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 8: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2800/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 00:12:32 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: -built arc
> The 'home' directory approach wouldn't help: with pre-built artifacts, it's
That does sound too many adjustments to make it work for 'home' directory approach.

I think you meant 'USE_PREBUILT_THIRDPARTY=0'


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh@50
PS11, Line 50: PREBUILT_URL
Is this a typo? Don't see this anywhere else.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh@51
PS11, Line 51: UPLOAD_PREBUILT_THIRDPARTY
What is the purpose of this if upload is not happening automatically?


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@80
PS11, Line 80: cut -d. -f1
Wouldn't this just be major version? (comment says major and minor version are used)
Same goes for below at line:91

Could be leftover typo in the comment from previous PS where both were considered?


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@107
PS11, Line 107: $CC
Should this be $compiler ?


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@266
PS11, Line 266: prepare_staging_prefix
This seems unused. fetch_prebuilt_or_build is doing this already.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@337
PS11, Line 337: INSTALL_DESTDIR="$TP_STAGING_DIR/$component"
Should this be null if USE_PREBUILT_THIRDPARTY=0 ?


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@389
PS11, Line 389: rm -rf "$staging_root"
Is a similar check (from prepare_staging_prefix) required before removing the directory?
++
if [ -z "$staging_root" ]; then
    echo "ERROR: staging directory must not be empty" >&2
    exit 1
fi
++



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 09:22:34 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2790/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 06:09:04 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 5:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@30
PS5, Line 30: After that, it's
            : possible to upload the result into a sub-directory of the designated
            : S3 bucket s3://cloudera-thirdparty-libs/prebuilt
> Question: To leverage this gain, how do we manage the manual upload of pre-
As of current state of this, I logged into the jenkins slave nodes and uploaded the newly built archives from there.  It's all done in one command line once proper creds are put into the environment.

In the long run, I'm planning to take care of upload in a separate changelist -- I don't want to mix it in with this initial update that focuses on providing the base for reducing 3rd-party build times.


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@35
PS5, Line 35: distrute
> distribute
Done


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: /opt/kudu/
The 'home' directory approach wouldn't help: with pre-built artifacts, it's necessary to guarantee same path at any machine, while PREFIX-related paths are hard-coded into the binaries and pkgconfig files.   If not using same layout for the 3rd-party artifacts everywhere, it'd necessary to  always override location of the libraries with LD_LIBRARY_PATH for every invocation of Kudu binaries and tests in runtime, add extra compilation flags, and also do 'manual' update of pkgconfig and other 3rd-party files upon installation.  I tried that and it turned to be too involved and very error-prone.

> I am guessing single user CI/dev system is the general use case here and concurrent build by multiple users is not something that needs to be considered here?

That's the only relevant and the most prevalent use case as of time of writing this, correct.

Concurrent builds by multiple users is OK if all those users utilize the same version of Kudu 3rd-party.  Concurrent builds might end up with links in /opt/kudu/thirdparty pointing to different target directories, but with same versions everywhere it's not a problem.

If it's desired to use different 3rd-party versions for different workspaces, the solution is simple: add `export USE_PREBUILT_THIRDPARTY=1` into .bashrc or .profile, and build everything from source in the local workspace.


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: uninstrumeted
> uninstrumented
Done


http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@62
PS5, Line 62: Only CMake/GNUmake combination is now used for building
            : 3rd-party components that rely on CMake to bootstrap build environment
> Is there a significant increase in build time if not using ninja, especiall
No, I haven't seen any significant increase in total build times for 3rd-party if not using ninja.  While using ninja provides a marginal improvement in build times, it's main appeal is a more compact output.

I guess a measured reduction in build times using ninja vs make might be seen if we weren't explicitly setting '-j' option for GNU make or having multiple build-recipe-heavy projects like LLVM, but that's not the case here.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 18:08:20 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#7).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 981 insertions(+), 349 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#10).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,025 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/10
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................


Patch Set 1:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2750/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 15 Jul 2026 06:30:38 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 4:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2763/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Jul 2026 16:41:59 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 4:

Build Aborted 

http://jenkins.kudu.apache.org/job/pre_commit/2763/ : ABORTED


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Jul 2026 04:42:02 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG@16
PS4, Line 16: * name of the component (cmake, protobuf, etc.)
: * version of the component as in $KUDU_HOME/thirdparty/vars.sh
: * OS name/flavor (redhat, ubuntu, macos, etc.)
: * OS major and minor version (9.2, 24.04, 26.5, etc.)
: * CPU architecture (x86_64, aarch64, etc.)
: * build toolchain info:
: ** compiler family (gcc, clang, etc.)
: ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)
There can be dependencies within the toolchain. For example, curl depends on zlib. If you have an identifier for curl that doesn't know about zlib, then you can end up in a situation where you're using a curl built against zlib X when you're now using zlib Y. With zlib, that probably isn't a problem, but it's a more general theme: Do the identifiers know about transitive dependencies?

In Impala's toolchain, we have a variety of these dependencies. Thrift depends on zlib and boost. googletcmalloc and re2 depend on abseil-cpp. orc depends on lz4, protobuf, snappy, zlib, zstd. Etc. A lot of these interfaces are stable, but some are not (e.g. abseil-cpp). We generate the toolchain using a Jenkins job and use the build number as an additional piece of the key. For developers, we have logic to avoid redownloading a package if the version is the same (i.e. if both the old and new build numbers have foo-x.y.z, we don't redownload it). If something goes wrong, someone can delete their toolchain and redownload a coherent copy. The cost is that each build is maintaining its own copy, so our toolchain footprint in the bucket is very large.

Separately, matching on a major+minor version can make it harder to populate the cache and get cache hits. For some cases, there aren't other minor versions (e.g. Ubuntu 22.04), but others have several (e.g. Redhat). For Redhat, Impala uses only the major version and builds on a single configuration for all versions (Rocky 9.2). That relies on the newer minor versions being backwards compatible.

http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh@163
PS4, Line 163: # Building a variable in the xxx_NAME form: vars.sh contains corresponding
: # versioned strings for all the 3rd-party components.
: local version_var="${name}_VERSION"
: local version="${!version_var}"
: if [ -z "$version" ]; then
: echo "ERROR: $version_var isn't defined: check vars.sh" >&2
: exit 1
: fi
: echo ${component_name}-${version}
Does this handle applying an additional patch to a component (or changing its build definition slightly)? For Impala, we have a patch level in the version number.

--
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 20 Jul 2026 20:45:39 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 12: Verified+1

RELEASE, TSAN: failures of unrelated flaky tests


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 01:19:23 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 14: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2819/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 05:38:22 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 13:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2817/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 19:18:51 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 13: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2817/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 20:16:51 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.

Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 14:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh@172
PS14, Line 172:   local plevel="${!plevel_var}"
nit: `$version` is validated just above (errors if empty), but `$plevel`
isn't. If a future component gets a `xxx_VERSION` but no `xxx_PATCHLEVEL`, this
silently emits `name-version.p` (empty patch level) and we'd fetch/create a
mis-named tarball rather than failing fast. Worth an `[ -z "$plevel" ]` guard
symmetric with the version check?


http://gerrit.cloudera.org:8080/#/c/24583/14/thirdparty/prebuilt-utils.sh@330
PS14, Line 330:   return 0
This `return 0` (combined with the callers at lines 369 and 410 invoking
this function as `if ! invoke_component_build ...`) silently swallows build
failures. When a function is run in an `if !` condition, bash disables `set -e`
for the *entire* call chain, so a failing `make` / `make install` inside
`"$build_func"` does not abort; execution falls through to the unconditional
`return 0`, which reports success. I verified this:

  set -e
  f() { false; echo reached; return 0; }
  if ! f; then echo err; fi   # prints "reached", never "err", exit 0

Consequences in the build-from-source path of fetch_prebuilt_or_build: a broken
build is treated as success, we then run create_prebuilt_tarball over an
incomplete/empty $staging_root (tar succeeds), cache a corrupt tarball, extract
it, and continue. Even a clean nonzero exit from the build func is masked by the
trailing `return 0`. Can we drop `return 0` (let the function's status
propagate) and, more importantly, avoid calling it under `if !` so `set -e`
stays active during the component build? e.g. call it directly and check `$?`,
or wrap the build in a subshell that re-enables `set -e`.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 11:45:49 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 15:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2828/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 21:16:25 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has removed a vote on this change.

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG@16
PS4, Line 16:   * name of the component (cmake, protobuf, etc.)
            :   * version of the component as in $KUDU_HOME/thirdparty/vars.sh
            :   * OS name/flavor (redhat, ubuntu, macos, etc.)
            :   * OS major and minor version (9.2, 24.04, 26.5, etc.)
            :   * CPU architecture (x86_64, aarch64, etc.)
            :   * build toolchain info:
            :     ** compiler family (gcc, clang, etc.)
            :     ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)
> Do the identifiers know about transitive dependencies?
Nope, they don't.  The idea is to rely on the dependencies as they are implicitly encoded in the build scripts/function in the thridparty of corresponding components.  If all the components are built using one particular dependency chain, this works as expected.

If the dependency chain changes, then it will be necessary to bump the versions of the components.  Alternatively, we can introduce some extra suffix or tag if there is a necessity to maintain versions of particular dependency lineages.


http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh@163
PS4, Line 163:   # Building a variable in the xxx_NAME form: vars.sh contains corresponding
             :   # versioned strings for all the 3rd-party components.
             :   local version_var="${name}_VERSION"
             :   local version="${!version_var}"
             :   if [ -z "$version" ]; then
             :     echo "ERROR: $version_var isn't defined: check vars.sh" >&2
             :     exit 1
             :   fi
             :   echo ${component_name}-${version}
> Does this handle applying an additional patch to a component (or changing i
If the patch version is reflected in the value of the corresponding xxx_VERSION variable that's defined in $KUDU_HOME/thirdparty/vars.sh (e.g., GTEST_VERSION), then yes -- this does handle it.



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Jul 2026 20:26:34 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#5).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distrute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumeted,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
8 files changed, 962 insertions(+), 339 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3: -Verified

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2762/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Jul 2026 03:02:29 +0000
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................


Patch Set 1: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2750/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 16 Jul 2026 00:27:03 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when it doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JARs and similar content
    (e.g., interpreted scripts, etc.).

CLEAN_THIRDPARTY

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Reviewed-on: http://gerrit.cloudera.org:8080/24583
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Marton Greber <gr...@gmail.com>
Reviewed-by: Ashwani Raina <ar...@gmail.com>
---
M CMakeLists.txt
M build-support/run_dist_test.py
M docker/Dockerfile
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
10 files changed, 1,016 insertions(+), 394 deletions(-)

Approvals:
  Alexey Serbin: Verified
  Marton Greber: Looks good to me, approved
  Ashwani Raina: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 16
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11:

(6 comments)

> (6 comments)
 > 
 > Thank you for addressing the comments.
 > 
 > Almost there. Just a few nits.
 > 
 > I am not sure if docker builds are automatically tested by
 > dist-cluster or has to be done explicitly. Would be great if you
 > could verify that everything checks out for docker build. Btw, I am
 > fine with addressing any unforeseen issues there in a separate
 > change.

Yep, I saw that AI tools provided code review feedback on so-called Docker build when asked for review, but I'm not sure that's relevant at all.  If anything comes up, I'm happy to address that in a follow-up changelist -- it's too much packed here already, and I was even thinking to start splitting this patch into pieces :)

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@80
PS11, Line 80: cut -d. -f1
> I may have highlighted wrong place that caused the confusion. I should have
Done


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@116
PS12, Line 116: $version_str
> Should this be in quotes to avoid collapse of newlines to spaces and always
Good catch -- indeed, quotes were missing here.


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@253
PS12, Line 253: Creating pre-b
> This is unused, can be removed here and in callers.
Done


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@381
PS12, Line 381: 
> s/$tarball/$tarball_name ?
Done


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@388
PS12, Line 388: # Cleanup the staging sub-direc
> $staging_root will always be non-empty with '/' lying around even if $compo
Done


http://gerrit.cloudera.org:8080/#/c/24583/12/thirdparty/prebuilt-utils.sh@413
PS12, Line 413: tall_sub
> s/$tarball/$tarball_name ?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 18:49:00 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 14: Code-Review+2

Looks good to me after addressing pending feedback.


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 13:14:21 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#14).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when it doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JARs and similar content
    (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,036 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/14
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 14
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11: Verified+1

Unrelated failures of known flaky tests in TSAN build: KUDU-3598, KUDU-3605, etc.


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 05:16:51 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 11:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45
PS5, Line 45: -built arc
> That does sound too many adjustments to make it work for 'home' directory a
Ah, right -- in my response I rather meant 'USE_PREBUILT_THIRDPARTY=0', yes.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh@50
PS11, Line 50: PREBUILT_URL
> Is this a typo? Don't see this anywhere else.
Good catch -- it's no longer used: now PREBUILT_THIRDPARTY_URL is used instead (see prebuilt-utils.sh).


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/build-thirdparty.sh@51
PS11, Line 51: UPLOAD_PREBUILT_THIRDPARTY
> What is the purpose of this if upload is not happening automatically?
It was here in a few very first revisions (not published in gerrit, though), but I removed it later.  This is a remnant of that; it's fixed now.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh
File thirdparty/prebuilt-utils.sh:

http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@80
PS11, Line 80: cut -d. -f1
> Wouldn't this just be major version? (comment says major and minor version 
As of PS5, it's only the major version, correct.  That's reflected in the commit description as well.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@107
PS11, Line 107: $CC
> Should this be $compiler ?
Done


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@266
PS11, Line 266: prepare_staging_prefix
> This seems unused. fetch_prebuilt_or_build is doing this already.
Good catch -- indeed, this isn't used anymore.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@337
PS11, Line 337: INSTALL_DESTDIR="$TP_STAGING_DIR/$component"
> Should this be null if USE_PREBUILT_THIRDPARTY=0 ?
Indeed: this should logically mirror the setting of the PREFIX_xxx variables in vars.sh.

It seems I missed verifying how this works with USE_PREBUILT_THIRDPARTY=0 after recent modifications.  Will do and update the rest of this patch as necessary.


http://gerrit.cloudera.org:8080/#/c/24583/11/thirdparty/prebuilt-utils.sh@389
PS11, Line 389: rm -rf "$staging_root"
> Is a similar check (from prepare_staging_prefix) required before removing t
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 22:58:12 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 12: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2809/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 23:53:44 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#11).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,022 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/11
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 6:

Build Aborted 

http://jenkins.kudu.apache.org/job/pre_commit/2803/ : ABORTED


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Jul 2026 03:23:31 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#8).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 981 insertions(+), 349 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/8
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#9).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major version (9, 24, 26, etc.) -- this assumes the newer minor
    versions of the same major release are backwards compatible
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler family (gcc, clang, etc.)
    ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distribute only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:

  aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,025 insertions(+), 393 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/9
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] WIP: KUDU-3788 an option to use pre-built thirdparty components

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: WIP: KUDU-3788 an option to use pre-built thirdparty components
......................................................................


Patch Set 2:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2757/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jul 2026 03:04:12 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24583

to look at the new patch set (#3).

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................

KUDU-3788 an option to use pre-built thirdparty artifacts

With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.

The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
  * name of the component (cmake, protobuf, etc.)
  * version of the component as in $KUDU_HOME/thirdparty/vars.sh
  * OS name/flavor (redhat, ubuntu, macos, etc.)
  * OS major and minor version (9.2, 24.04, 26.5, etc.)
  * CPU architecture (x86_64, aarch64, etc.)
  * build toolchain info:
    ** compiler name (gcc, clang)
    ** C++ compiler version in form '<major>' or '<major>.<minor>'
       if the minor version information is available, as reported by
       '<compiler> -dumpversion'

The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature.  If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache.  After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt.  I didn't implement
automatic uploading into the bucket because:
  * it makes sense to limit upload and refresh of the pre-built
    archives, making sure we distrute only only vetted and verified ones
  * as for CI, our current upstream CI pipeline isn't supposed to keep
    valid S3 credentials in the build environment

The prefix is set to /opt/kudu/thirdparty/{common,uninstrumeted,tsan}
when building pre-built artifacts for a 3rd-party component.  This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach.  This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed.  When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.

Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available.  Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.

Below a list of TODOs to address in follow-up changelists:
  * address the issue with trace-viewer: it should be packed into
    pre-built archive as well, and links should be established
    from $KUDU_HOME/www into somewhere under thirdparty/installed
    instead of copying files directly into $KUDU_HOME
  * avoid unconditionally downloading and unpacking source tarballs:
    as of this version, the build script is still downloading and
    unpacking the source archives for all the 3rd-party components
    when doesn't find corresponding sub-directory in
    $KUDU_HOME/thirdparty/src
  * add CPU features that are auto-detected and enabled for building
    particular 3rd-party components, e.g., AVX2 instructions support
    when building bitshuffle
  * introduce matching of any CPU architecture and any OS: this
    should help with de-duplication of packed JAR content and similar
    content (e.g., interpreted scripts, etc.).

Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
---
M CMakeLists.txt
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
8 files changed, 948 insertions(+), 339 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/24583/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 3: -Verified

Build Started http://jenkins.kudu.apache.org/job/pre_commit/2761/


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Jul 2026 01:18:32 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3788 an option to use pre-built thirdparty artifacts

Posted by "Ashwani Raina (Code Review)" <ge...@cloudera.org>.

Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )

Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................


Patch Set 15: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Gerrit-Change-Number: 24583
Gerrit-PatchSet: 15
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 03 Aug 2026 07:27:06 +0000
Gerrit-HasComments: No