You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org> on 2024/10/02 17:46:31 UTC

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21873


Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................

KUDU-3604: Pin Cython Version, Package Maintenance

Currently, users need to manually install Cython before installing the
Kudu Python package. By running '$ pip install Cython' on most
relatively new systems, they obtain Cython 3.x. However, our setup.py
and build process are not compatible with Cython 3.x. In this patch,
I've added a prerequisite step to setup.py that identifies and
installs the correct Cython version before all the package build/install
steps.

Additionally, the Kudu Python package pulled from PyPi includes all the
test files. This was resolved with a simple change in MANIFEST.in to
exclude the test source folder. Moreover, end-users were also getting
the test dependencies from setup.py because we had a single
requirements.txt file for both distribution and development. I have
now split the dependencies into two files: requirements.txt and
requirements_dev.txt. The dependencies in requirements.txt are
automatically pulled during the execution of setup.py. For development
tasks, developers need to manually install requirements_dev.txt before
running setup.py. The build-and-test.sh script was also updated to
use requirements_dev.txt.

Furthermore, there were several details and edge cases covered in the CI
script related to older OS versions (e.g., el6) and unsupported Python
versions (e.g., Python 2.6), as well as unpinned package versions. Since
all packages are now version-pinned and these limitations are no longer
relevant, I have cleaned up the package installation part in
build-and-test.sh for the Python packages.

As a follow-up patch, I plan to add more developer-facing documentation
on how to set up the development environment, run unit tests, debug,
etc.

Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Reviewed-on: http://gerrit.cloudera.org:8080/21816
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Tested-by: Abhishek Chennaka <ac...@cloudera.com>
(cherry picked from commit 8d9b3574a12bb98b79e29375c76f98a1a48af5ad)
---
M build-support/jenkins/build-and-test.sh
M python/MANIFEST.in
M python/README.md
M python/requirements.txt
A python/requirements_dev.txt
M python/setup.py
A python/setup_prerequisites.py
7 files changed, 94 insertions(+), 143 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/73/21873/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has removed a vote on this change.

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21873 )

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................

KUDU-3604: Pin Cython Version, Package Maintenance

Currently, users need to manually install Cython before installing the
Kudu Python package. By running '$ pip install Cython' on most
relatively new systems, they obtain Cython 3.x. However, our setup.py
and build process are not compatible with Cython 3.x. In this patch,
I've added a prerequisite step to setup.py that identifies and
installs the correct Cython version before all the package build/install
steps.

Additionally, the Kudu Python package pulled from PyPi includes all the
test files. This was resolved with a simple change in MANIFEST.in to
exclude the test source folder. Moreover, end-users were also getting
the test dependencies from setup.py because we had a single
requirements.txt file for both distribution and development. I have
now split the dependencies into two files: requirements.txt and
requirements_dev.txt. The dependencies in requirements.txt are
automatically pulled during the execution of setup.py. For development
tasks, developers need to manually install requirements_dev.txt before
running setup.py. The build-and-test.sh script was also updated to
use requirements_dev.txt.

Furthermore, there were several details and edge cases covered in the CI
script related to older OS versions (e.g., el6) and unsupported Python
versions (e.g., Python 2.6), as well as unpinned package versions. Since
all packages are now version-pinned and these limitations are no longer
relevant, I have cleaned up the package installation part in
build-and-test.sh for the Python packages.

As a follow-up patch, I plan to add more developer-facing documentation
on how to set up the development environment, run unit tests, debug,
etc.

Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Reviewed-on: http://gerrit.cloudera.org:8080/21816
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Tested-by: Abhishek Chennaka <ac...@cloudera.com>
(cherry picked from commit 8d9b3574a12bb98b79e29375c76f98a1a48af5ad)
Reviewed-on: http://gerrit.cloudera.org:8080/21873
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M build-support/jenkins/build-and-test.sh
M python/MANIFEST.in
M python/README.md
M python/requirements.txt
A python/requirements_dev.txt
M python/setup.py
A python/setup_prerequisites.py
7 files changed, 94 insertions(+), 143 deletions(-)

Approvals:
  Alexey Serbin: Looks good to me, approved
  Abhishek Chennaka: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21873 )

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................


Patch Set 1: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/556/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 19:56:06 +0000
Gerrit-HasComments: No

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21873 )

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................


Patch Set 1: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 18:11:50 +0000
Gerrit-HasComments: No

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21873 )

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Oct 2024 17:46:03 +0000
Gerrit-HasComments: No

[kudu-CR](branch-1.18.x) KUDU-3604: Pin Cython Version, Package Maintenance

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21873 )

Change subject: KUDU-3604: Pin Cython Version, Package Maintenance
......................................................................


Patch Set 1:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/556/


-- 
To view, visit http://gerrit.cloudera.org:8080/21873
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Gerrit-Change-Number: 21873
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 17:46:39 +0000
Gerrit-HasComments: No