You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/09/07 23:52:37 UTC

[native-toolchain-CR] [tools] Install aws to Python 3 when available

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20465


Change subject: [tools] Install aws to Python 3 when available
......................................................................

[tools] Install aws to Python 3 when available

I encountered failures building build containers for Ubuntu 22.04, and
CentOS 8 on ARM, when installing awscli. We should start moving to
Python 3 anyway, so this patch installs awscli to Python 3 when it
detects pip3 is available. Other platforms will continue to install pip
and use Python 2.7.

Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
---
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/ubuntu2204.df
3 files changed, 8 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/20465/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................

IMPALA-12435: Install pinned pip

Fixes installing the desired version of pip. Pulling a specific version
of the get-pip script is insufficient to install that version of pip.
Installing awscli started failing due to config deprecated in later
versions of pip.

Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Reviewed-on: http://gerrit.cloudera.org:8080/20465
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
2 files changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Michael Smith: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................


Patch Set 3:

> > Patch Set 3: Code-Review+2
 > >
 > > Thanks for fixing this
 > >
 > > Does this help with the ARM container builds?
 > 
 > No, it runs into a different build failure in ARM containers.
 > Planning to move to Python 3.8 for them.

Darn, well at least we fixed this piece.


-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 22:26:22 +0000
Gerrit-HasComments: No

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................


Patch Set 3: Verified+1

All images but SLES 12 built successfully. Ran into an unrelated error with SLES 12 trying to pull opensuse.org repos.


-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 23:09:32 +0000
Gerrit-HasComments: No

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................


Patch Set 3: Code-Review+2

Thanks for fixing this

Does this help with the ARM container builds?


-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 22:19:57 +0000
Gerrit-HasComments: No

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20465/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20465/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-12435: Install pinned pip
> Please file a JIRA and then also give a bit of info about the error seen. I
Done


http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh
File docker/all/postinstall.sh:

http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh@65
PS1, Line 65:     dl_verify https://r
> This will not remain constant over time. Is there a specific version that w
There's no single version that works. I decided to take this change out for now.


http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh@72
PS1, Line 72: 
> I noticed that this command doesn't install 20.3.4 like I thought it did. O
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 18:52:51 +0000
Gerrit-HasComments: Yes

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20465

to look at the new patch set (#3).

Change subject: IMPALA-12435: Install pinned pip
......................................................................

IMPALA-12435: Install pinned pip

Fixes installing the desired version of pip. Pulling a specific version
of the get-pip script is insufficient to install that version of pip.
Installing awscli started failing due to config deprecated in later
versions of pip.

Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
---
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/20465/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[native-toolchain-CR] IMPALA-12435: Install pinned pip, use pip3 when available

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20465

to look at the new patch set (#2).

Change subject: IMPALA-12435: Install pinned pip, use pip3 when available
......................................................................

IMPALA-12435: Install pinned pip, use pip3 when available

Fixes installing the desired version of pip. Pulling a specific version
of the get-pip script is insufficient to install that version of pip.
Installing awscli started failing due to config deprecated in later
versions of pip.

Also moves to installing awscli to Python 3 when it detects pip3 is
available. Other platforms will continue to install pip and use Python
2.7.

Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
---
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/ubuntu2204.df
3 files changed, 9 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/20465/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[native-toolchain-CR] IMPALA-12435: Install pinned pip

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: IMPALA-12435: Install pinned pip
......................................................................


Patch Set 3:

> Patch Set 3: Code-Review+2
> 
> Thanks for fixing this
> 
> Does this help with the ARM container builds?

No, it runs into a different build failure in ARM containers. Planning to move to Python 3.8 for them.


-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 22:20:25 +0000
Gerrit-HasComments: No

[native-toolchain-CR] [tools] Install aws to Python 3 when available

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: [tools] Install aws to Python 3 when available
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh
File docker/all/postinstall.sh:

http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh@72
PS1, Line 72:     python get-pip.py
I noticed that this command doesn't install 20.3.4 like I thought it did. On my box, changing this to force pip==20.3.4 makes the Ubuntu 22 image build on x86:
python get-pip.py "pip==20.3.4"



-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 00:36:59 +0000
Gerrit-HasComments: Yes

[native-toolchain-CR] [tools] Install aws to Python 3 when available

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20465 )

Change subject: [tools] Install aws to Python 3 when available
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20465/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20465/1//COMMIT_MSG@7
PS1, Line 7: [tools] Install aws to Python 3 when available
Please file a JIRA and then also give a bit of info about the error seen. It looks like Ubuntu 22 is failing on x86, so this is just a generic issue not specific to ARM.


http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh
File docker/all/postinstall.sh:

http://gerrit.cloudera.org:8080/#/c/20465/1/docker/all/postinstall.sh@65
PS1, Line 65:     pip3 install awscli
This will not remain constant over time. Is there a specific version that works?



-- 
To view, visit http://gerrit.cloudera.org:8080/20465
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4667181f9a6ef66c757623f1b128c3faa2057114
Gerrit-Change-Number: 20465
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 00:26:28 +0000
Gerrit-HasComments: Yes