You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "danepitkin (via GitHub)" <gi...@apache.org> on 2023/08/09 20:30:25 UTC
[GitHub] [arrow] danepitkin opened a new pull request, #37097: WIP: [Python] Add support for Cython 3.0.0
danepitkin opened a new pull request, #37097:
URL: https://github.com/apache/arrow/pull/37097
Further testing of https://github.com/apache/arrow/pull/36745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684358488
I'm also not sure how to fix these new numpydoc lint issues:
```
INFO:archery:Running Python docstring linters
pyarrow._s3fs.S3LogLevel
-> pyarrow._s3fs.An enumeration.
PR01: Parameters {'qualname', 'module', 'names', 'start', 'type', 'value'} not documented
pyarrow._fs.FileType
-> pyarrow._fs.An enumeration.
PR01: Parameters {'qualname', 'module', 'names', 'start', 'type', 'value'} not documented
pyarrow.lib.MetadataVersion
-> pyarrow.lib.An enumeration.
PR01: Parameters {'qualname', 'module', 'names', 'start', 'type', 'value'} not documented
pyarrow._dataset.type.FileSystemDataset.from_paths
-> pyarrow._dataset.FileSystemDataset.from_paths(cls, paths, schema=None, format=None, filesystem=None, partitions=None, root_partition=None)
PR01: Parameters {'schema', 'filesystem', 'format', 'root_partition', 'partitions'} not documented
pyarrow.lib.MetadataVersion
-> pyarrow.lib.An enumeration.
PR01: Parameters {'qualname', 'module', 'names', 'start', 'type', 'value'} not documented
Total number of docstring violations: 5
```
The enums are all defined in Cython with `cpdef`, which is a wrapper for https://peps.python.org/pep-0435/. Maybe cython or numpydoc isn't parsing correctly?
The `pyarrow._dataset.type.FileSystemDataset.from_paths` error seems like a bug in cython or numpydocs. The parameters are already documented and visible here https://arrow.apache.org/docs/python/generated/pyarrow.dataset.FileSystemDataset.html#pyarrow.dataset.FileSystemDataset.from_paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686960611
```
No such option: --no-cache
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/5930876428
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1683990371
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1308747539
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Those comment don't seem to silence the failure in the "Python / AMD64 Conda Python 3.9 Sphinx & Numpydoc (pull_request) " build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag`.
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` in cython 3, while they are of type `enum` in cython 2. When validating with numpydoc, the enums don't get parsed in cython 2. Here's the cython commit[1] that changes this behavior.
Cython 3:
```
# Python objects
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
Cython 2:
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<enum 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> type(pa.MetadataVersion.V5)
<enum 'MetadataVersion'>
```
[1] https://github.com/cython/cython/commit/74bcd91438a6f7f94417dfc4d50f572cfa50f820#diff-74af7ca253322fe3a89b0ad7062281cee4fafa00d868d4ff997fc9c924599836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1307622131
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1838,7 +1838,7 @@ cdef class FileFragment(Fragment):
typ = ""
partition_dict = get_partition_keys(self.partition_expression)
partition = ", ".join(
- [f"{key}={val}" for key, val in partition_dict.items()]
+ sorted([f"{key}={val}" for key, val in partition_dict.items()])
Review Comment:
I'll go ahead and remove sorting here and have the test handle non-deterministic repr output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1679001158
`test-cuda-python` and `test-ubuntu-20.04-python-3` are failing due to this cython issue https://github.com/cython/cython/issues/5552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1308733989
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1838,7 +1838,7 @@ cdef class FileFragment(Fragment):
typ = ""
partition_dict = get_partition_keys(self.partition_expression)
partition = ", ".join(
- [f"{key}={val}" for key, val in partition_dict.items()]
+ sorted([f"{key}={val}" for key, val in partition_dict.items()])
Review Comment:
Sorry, forgot to answer here: I somehow thought this was the path, but it's a separate list of partition key/values. For those I suppose the order is indeed not strictly important.
But since you already updated the test to deal with it in the meantime, let's leave it at that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1307620786
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
I'll see if I can fix this in a separate PR first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1721580659
Revision: 6f32fcf3e02aec40d6bf9cbebbf39b4c54091093
Submitted crossbow builds: [ursacomputing/crossbow @ actions-165015566a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-165015566a)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/tree/actions-165015566a-github-test-conda-python-3.10)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6200743589/job/16836050547)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6200743093/job/16836049221)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6200735748/job/16836026069)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6200741527/job/16836043441)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6200740402/job/16836040492)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6200738398/job/16836034445)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6200741013/job/16836041819)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6200735222/job/16836024437)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6200737107/job/16836030164)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6200735010/job/16836023742)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6200734898/job/16836023274)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6200734724/job/16836022793)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6200738799/job/16836035539)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6200736081/job/16836027202)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6200739324/job/16836037007)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6200737988/job/16836032921)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6200737601/job/16836031596)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6200743811/job/16836051349)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6200736598/job/16836028764)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16836049570)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16836040467)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16836050999)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6200742630/job/16836047591)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1328724637
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
Can you just add a comment pointing to GH-37235?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729209013
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1683996134
Revision: 9aeefc5f9e4dbd9c5510c7b2f38458956d2c0170
Submitted crossbow builds: [ursacomputing/crossbow @ actions-d1b1f00367](https://github.com/ursacomputing/crossbow/branches/all?query=actions-d1b1f00367)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/5903743048/job/16014325078)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/5903746902/job/16014336298)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/5903750452/job/16014347240)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/5903736018/job/16014304427)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/5903746264/job/16014334414)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5903748744/job/16014341779)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5903747482/job/16014338393)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/5903740911/job/16014318891)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5903755277/job/16014362593)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/5903754034/job/16014358533)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/5903751526/job/16014350765)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5903749882/job/16014345519)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/5903742398/job/16014323116)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/5903744940/job/16014330643)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/5903735758/job/16014303792)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/5903744289/job/16014328932)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/5903738762/job/16014312484)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/5903752811/job/16014354584)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5903752184/job/16014352795)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/5903738110/job/16014310675)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5903741716/job/16014321084)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/5903753332/job/16014356432)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16014338923)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16014334232)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16014362491)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/5903743664/job/16014326936)|
|verify-rc-source-python-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903737463/job/16014308618)|
|verify-rc-source-python-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903749295/job/16014343596)|
|verify-rc-source-python-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903740292/job/16014316864)|
|verify-rc-source-python-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903739595/job/16014314871)|
|verify-rc-source-python-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903736663/job/16014306217)|
|verify-rc-source-python-macos-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903748133/job/16014339986)|
|verify-rc-source-python-macos-conda-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5903751046/job/16014349158)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1301683589
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1838,7 +1838,7 @@ cdef class FileFragment(Fragment):
typ = ""
partition_dict = get_partition_keys(self.partition_expression)
partition = ", ".join(
- [f"{key}={val}" for key, val in partition_dict.items()]
+ sorted([f"{key}={val}" for key, val in partition_dict.items()])
Review Comment:
Yes, the `__repr__` test was failing. I thought it might be nice to have deterministic output for the `__repr__`. The key/value pair ordering was swapped when I compiled with Cython 3. Would it be better to update the test case to handle non-determinism?
https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_dataset.py#L1614
```
def test_fragments_repr(tempdir, dataset):
# partitioned parquet dataset
fragment = list(dataset.get_fragments())[0]
assert (
repr(fragment) ==
"<pyarrow.dataset.ParquetFileFragment path=subdir/1/xxx/file0.parquet "
"partition=[key=xxx, group=1]>"
)
```
My test output was returning `"partition=[group=1, key=xxx]>"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295847066
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -439,6 +438,28 @@ def allow_append_to_file(request, filesystem_config):
return filesystem_config['allow_append_to_file']
+@pytest.fixture(
+ params=[
+ pytest.lazy_fixture('builtin_pickle'),
+ pytest.lazy_fixture('cloudpickle')
+ ]
+)
+def pickle(request):
+ return request.param
Review Comment:
We have other pickling tests which could benefit from this. Perhaps share this fixture with all test modules and make sure it gets used everywhere possible?
(also perhaps rename it to `picklemodule` for less confusion, though YMMV)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1297718527
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Nevermind.. it is possible to make this a staticmethod. I don't know why I thought it wasn't working prior. Will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309361535
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Ugh, numpydoc doesn't parse `cdef` and `cpdef` tokens. Only `def` and `class`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag`.
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> from enum import Enum
>>>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` in cython 3, while they are of type `enum` in cython 2. When validating with numpydoc, the enums don't get parsed in cython 2.
Cython 3:
```
# Python objects
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
Cython 2:
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<enum 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> type(pa.MetadataVersion.V5)
<enum 'MetadataVersion'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` aka `enum.EnumType`.
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1699786039
Revision: abc3dba1063972d0dce40b7a789423eb0dd0a2b2
Submitted crossbow builds: [ursacomputing/crossbow @ actions-824b0285b7](https://github.com/ursacomputing/crossbow/branches/all?query=actions-824b0285b7)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/6029659259/job/16359705970)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/6029671944/job/16359744882)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/6029659585/job/16359707247)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6029661591/job/16359713363)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6029663968/job/16359720179)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6029662808/job/16359716948)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6029658178/job/16359702678)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6029671282/job/16359743050)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6029661037/job/16359711882)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6029668778/job/16359735766)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6029667136/job/16359730014)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6029672417/job/16359746229)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6029664525/job/16359721510)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6029660033/job/16359708747)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6029668354/job/16359734444)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6029670226/job/16359740112)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6029666702/job/16359728604)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6029673435/job/16359749853)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6029667527/job/16359731478)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6029670862/job/16359741920)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6029674268/job/16359752668)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6029663410/job/16359718672)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6029658854/job/16359704644)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16359749479)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16359717117)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/tree/actions-824b0285b7-azure-test-ubuntu-20.04-python-3)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6029669623/job/16359738372)|
|verify-rc-source-python-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029658522/job/16359703446)|
|verify-rc-source-python-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029669084/job/16359736822)|
|verify-rc-source-python-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029665034/job/16359723271)|
|verify-rc-source-python-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029667986/job/16359732996)|
|verify-rc-source-python-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029673843/job/16359750967)|
|verify-rc-source-python-macos-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029660628/job/16359710358)|
|verify-rc-source-python-macos-conda-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6029668774/job/16359735670)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1301509698
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1838,7 +1838,7 @@ cdef class FileFragment(Fragment):
typ = ""
partition_dict = get_partition_keys(self.partition_expression)
partition = ", ".join(
- [f"{key}={val}" for key, val in partition_dict.items()]
+ sorted([f"{key}={val}" for key, val in partition_dict.items()])
Review Comment:
Was this needed to get some test passing? It seems strange (or wrong?) that this needs to be sorted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1675270642
Potentially related cloudpickle issue? https://github.com/cloudpipe/cloudpickle/issues/506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1721977141
IMO this can be merged now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1310708573
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Yep, Enums are treated like classes (no special handling) https://github.com/numpy/numpydoc/issues/458
I'll just skip them in archery for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309260795
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
I'm still not able to reproduce these locally, but I did seem to add them to the wrong place. I updated the code and will test via CI jobs for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1699906941
@github-actions crossbow submit test-cuda-python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729154099
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1679055941
Current summary:
* Numpydocs now correctly reports 72 cython functions are missing documentation for their parameters (Should be fixed in a separate PR IMO)
* https://github.com/cython/cython/issues/5552 is breaking `test-cuda-python` and `test-ubuntu-20.04-python-3` nightly jobs
* Dask/Spark failures are on main and can be ignored here
Other than the above issues, I think this PR is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1297746837
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
Filed https://github.com/apache/arrow/issues/37235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686944825
Revision: dd5502533ca3e99e5906cb53ba90c525f788578d
Submitted crossbow builds: [ursacomputing/crossbow @ actions-25426b6d47](https://github.com/ursacomputing/crossbow/branches/all?query=actions-25426b6d47)
|Task|Status|
|----|------|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/5930798006/job/16081299309)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686940670
@github-actions crossbow submit test-conda-python-3.10-cython2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1328723772
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1075,10 +1075,11 @@ cdef class FileSystemDataset(Dataset):
self.partition_expression
)
- @classmethod
- def from_paths(cls, paths, schema=None, format=None,
- filesystem=None, partitions=None, root_partition=None):
- """A Dataset created from a list of paths on a particular filesystem.
+ @staticmethod
+ def from_paths(paths, schema=None, format=None, filesystem=None,
+ partitions=None, root_partition=None):
Review Comment:
Hmm... classmethod doesn't work correctly on Cython 3 anymore? Is it a known issue? Or is this change actually not needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729214951
Revision: 3c4f581103560cda3cf848c3e1b7ae3c86f1e19f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-d63bfc88b3](https://github.com/ursacomputing/crossbow/branches/all?query=actions-d63bfc88b3)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6259875548/job/16996758184)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6259881657/job/16996775057)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6259877264/job/16996764006)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6259878841/job/16996768337)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6259885629/job/16996787574)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6259879700/job/16996770613)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6259889528/job/16996798653)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6259887754/job/16996794039)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6259880772/job/16996773247)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6259891084/job/16996803305)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6259874931/job/16996755813)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6259886322/job/16996789936)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6259882439/job/16996777216)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6259890300/job/16996800876)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6259876165/job/16996760067)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6259886950/job/16996791628)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6259883262/job/16996779630)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6259874664/job/16996755304)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6259884854/job/16996784705)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6259876736/job/16996762155)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16996787980)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16996789962)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16996797612)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6259878350/job/16996767089)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` in cython 3, while they are of type `enum` in cython 2.
Cython 3:
```
# Python objects
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
Cython 2:
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<enum 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> type(pa.MetadataVersion.V5)
<enum 'MetadataVersion'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` in cython 3, while they are of type `enum` in cython 2.
Cython 3:
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
Cython 2:
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<enum 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> type(pa.MetadataVersion.V5)
<enum 'MetadataVersion'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag`.
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1716009372
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1291689805
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -1564,6 +1564,7 @@ def test_builtin_pickle_dataset(tempdir, datadir, use_legacy_dataset):
dataset, pickler=pickle, use_legacy_dataset=use_legacy_dataset)
+@pytest.mark.skip
Review Comment:
TODO: unskip test before merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1679599447
> Numpydocs now correctly reports 72 cython functions are missing documentation for their parameters (Should be fixed in a separate PR IMO)
Could you work on this before we merge this to avoid CI failures on main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684104526
Revision: 2c1f54a2f7b08ca5720b4e636f4e54022e991fcd
Submitted crossbow builds: [ursacomputing/crossbow @ actions-97bf744467](https://github.com/ursacomputing/crossbow/branches/all?query=actions-97bf744467)
|Task|Status|
|----|------|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5904522670/job/16016779126)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1298827598
##########
python/pyarrow/_s3fs.pyx:
##########
@@ -388,9 +388,10 @@ cdef class S3FileSystem(FileSystem):
FileSystem.init(self, wrapped)
self.s3fs = <CS3FileSystem*> wrapped.get()
- @classmethod
- def _reconstruct(cls, kwargs):
- return cls(**kwargs)
+ def _reconstruct(kwargs):
Review Comment:
TODO: I'm missing the `@staticmethod` keyword here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1297700420
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
+ return ParquetFragmentScanOptions(**kwargs)
Review Comment:
BTW, I did confirm this works on Cython 0.29.36
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296194400
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
IIRC I tried making a staticmethod defined within the class, but the docs recommended making a global static method and then adding it to the class with `no_method = staticmethod(_reconstruct)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296188601
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
This function executes user-defined python code, which absolutely can raise an exception. In fact, the test cases specifically _do_ raise an exception. The problem is that the C++ functionality explicitly chooses to ignore it and return its own error. The pyarrow binding is designed with this in mind, so adding `noexcept` is mimicking the expected behavior of previous cython versions. I'll file an issue to request refactoring of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1678155702
Revision: c8e6a19063768c7e6a08a2e7b878b1bb4f28a9cc
Submitted crossbow builds: [ursacomputing/crossbow @ actions-ba5c455e80](https://github.com/ursacomputing/crossbow/branches/all?query=actions-ba5c455e80)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/5861226686/job/15890904644)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/5861220955/job/15890890488)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/5861223853/job/15890897710)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/5861224194/job/15890898539)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/5861225231/job/15890900664)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5861220611/job/15890889321)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5861223476/job/15890896874)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/5861222429/job/15890895029)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5861227866/job/15890908498)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/5861221475/job/15890892254)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/5861225548/job/15890901566)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5861227281/job/15890906383)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/5861220792/job/15890889861)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/5861227423/job/15890906820)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/5861225766/job/15890902030)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/5861226949/job/15890905379)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/5861221099/job/15890891167)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/5861227735/job/15890907940)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/5861225965/job/15890902651)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/5861222029/job/15890893552)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/5861227610/job/15890907448)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/5861228352/job/15890910157)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/15890890471)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/15890901619)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/15890904529)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/5861222270/job/15890894235)|
|verify-rc-source-python-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861224918/job/15890899808)|
|verify-rc-source-python-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861221244/job/15890891587)|
|verify-rc-source-python-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861222617/job/15890895239)|
|verify-rc-source-python-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861221770/job/15890892861)|
|verify-rc-source-python-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861223115/job/15890896089)|
|verify-rc-source-python-macos-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861228186/job/15890909596)|
|verify-rc-source-python-macos-conda-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/5861228025/job/15890909057)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295836288
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Would it work if you make `_reconstruct` above a `staticmethod` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295173857
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
+ return ParquetFragmentScanOptions(**kwargs)
Review Comment:
That reminds me, I wanted to add a Cython<3 nightly job to CI to test future changes don't break anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #37097: GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1675389158
I've copied #36745 PR description.
We should update the PR description before we merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1308729971
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
> I'll see if I can fix this in a separate PR first.
It would certainly be nice to fix that, but FWIW I don't think that should hold up this PR even longer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag`.
```
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1701321326
Revision: 0aff6fdfd2dd9cb68640207fc4ba42819ba1a99f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f79b2bd742](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f79b2bd742)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/tree/actions-f79b2bd742-github-example-python-minimal-build-fedora-conda)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/6039726199/job/16389024442)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/6039713389/job/16388984463)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6039708447/job/16388972904)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6039715498/job/16388991287)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6039719608/job/16389004718)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6039711132/job/16388978861)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6039709579/job/16388975156)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6039721529/job/16389010314)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6039731483/job/16389040540)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6039722182/job/16389011909)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6039732313/job/16389043159)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6039712019/job/16388981194)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6039714803/job/16388988405)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6039730567/job/16389038126)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6039733230/job/16389045800)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6039723083/job/16389014349)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6039717941/job/16388999686)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6039727631/job/16389029470)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6039720541/job/16389007332)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6039723846/job/16389016645)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6039714210/job/16388986409)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6039716689/job/16388995295)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16389032674)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16388995602)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16389014932)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6039729741/job/16389035790)|
|verify-rc-source-python-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039709015/job/16388974004)|
|verify-rc-source-python-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039721318/job/16389009840)|
|verify-rc-source-python-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039725484/job/16389022111)|
|verify-rc-source-python-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039724482/job/16389018997)|
|verify-rc-source-python-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039707954/job/16388972047)|
|verify-rc-source-python-macos-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039717254/job/16388997267)|
|verify-rc-source-python-macos-conda-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6039718803/job/16389002357)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729774103
None of the failures look related, wil merge. Thanks again Dane!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1696441189
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1697386541
> > `test-cuda-python` and `test-ubuntu-20.04-python-3` are failing due to this cython issue [cython/cython#5552](https://github.com/cython/cython/issues/5552)
>
> Those are still failing. What do we do about those? Can we pin cython to cython<3 on those builds for now? Or disable gdb support on those builds?
Good question. Hmm. These are great suggestions. I like the idea of pinning to cython<3 so one can still use gdb. Maybe I can attempt the cython fix later on (and I can file an issue in Arrow to unpin cython for these builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1687084759
@github-actions crossbow submit test-conda-python-3.10-cython2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1301683589
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1838,7 +1838,7 @@ cdef class FileFragment(Fragment):
typ = ""
partition_dict = get_partition_keys(self.partition_expression)
partition = ", ".join(
- [f"{key}={val}" for key, val in partition_dict.items()]
+ sorted([f"{key}={val}" for key, val in partition_dict.items()])
Review Comment:
Yes, the `__repr__` test was failing. I thought it might be nice to have deterministic output for the `__repr__`. The key/value pair ordering was swapped when I compiled with Cython 3. Would it be better to update the test case to handle non-determinism?
https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_dataset.py#L1614
```
def test_fragments_repr(tempdir, dataset):
# partitioned parquet dataset
fragment = list(dataset.get_fragments())[0]
assert (
repr(fragment) ==
"<pyarrow.dataset.ParquetFileFragment path=subdir/1/xxx/file0.parquet "
"partition=[key=xxx, group=1]>"
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686959436
@github-actions crossbow submit test-conda-python-3.10-cython2 --no-cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686965464
@github-actions archery docker run --no-leaf-cache conda-python-cython2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1310700873
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
> BTW, on the latest commit, `FileSystemDataset.from_paths` isn't erroring anymore, so maybe the change from classmethod to staticmethod "fixed" it ..
I noticed that, too! There's some weird stuff going on here with parsing. This change shouldn't have mattered.
For the enums, it seems numpydoc just doesn't parse properly. It's getting the Enum class function signature, but it should really be documenting the values. I don't see any reference to the word "enum" in numpydocs documentation so I'll assume it is unsupported. I can skip enum validation in archery numpydoc.
```
>>> inspect.signature(pa.MetadataVersion)
<Signature (value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)>
>>> inspect.signature(Flag)
<Signature (value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1699909437
Revision: c9f3f17f490c8ff681ce5cbc48495e641b189d06
Submitted crossbow builds: [ursacomputing/crossbow @ actions-81a6076e9f](https://github.com/ursacomputing/crossbow/branches/all?query=actions-81a6076e9f)
|Task|Status|
|----|------|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6030510024/job/16362480505)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1716010971
Rebased. I am going to wait for Cython 3.0.3, which will fix the `cython --gdb` compilation issue seen in some of the nightly builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295836859
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Or it might also be possible to disable the `binding` option specifically for `_reconstruct`:
https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295844513
##########
python/pyarrow/scalar.pxi:
##########
@@ -809,8 +809,8 @@ cdef class MapScalar(ListScalar):
Iterate over this element's values.
"""
arr = self.values
- if array is None:
- raise StopIteration
+ if arr is None:
+ return
Review Comment:
Nice catch... I guess this code path is not tested? Can you add a test for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684349408
I'd like to merge https://github.com/apache/arrow/pull/37255 and rebase to make this PR cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1298761676
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Ahh I remember what happened. This works for Cython 3, but not Cython 0.29.36.
With Cython 0.29.36, we just need to add the `binding=True` compiler option.
my_test.pyx
```
# cython : language_level = 3
import cython
cdef class Foo():
@staticmethod
@cython.binding(True) # This works
def _reconstruct():
return Foo()
def __reduce__(self):
return Foo._reconstruct, tuple()
@staticmethod
def _reconstruct(x):
return Bar(x)
cdef class Bar():
@staticmethod # This does not work
def _reconstruct():
return Bar()
def __reduce__(self):
return Bar._reconstruct, tuple()
```
Usage here:
```
In [1]: from my_test import Foo, Bar
In [2]: f = Foo()
In [3]: b = Bar()
In [4]: import pickle
In [5]: pickle.dumps(f)
Out[5]: b'\x80\x04\x95#\x00\x00\x00\x00\x00\x00\x00\x8c\x07my_test\x94\x8c\x10Foo._reconstruct\x94\x93\x94)R\x94.'
In [6]: pickle.dumps(b)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call last)
Cell In[6], line 1
----> 1 pickle.dumps(b)
PicklingError: Can't pickle <built-in function _reconstruct>: it's not the same object as my_test._reconstruct
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295842877
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
Are we sure this function cannot raise a Python exception? Its code is definitely non-trivial, so the "solution" here looks more like a bandaid.
In the interest of moving this forward, can you open a bug for this and we'll revisit later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1298827598
##########
python/pyarrow/_s3fs.pyx:
##########
@@ -388,9 +388,10 @@ cdef class S3FileSystem(FileSystem):
FileSystem.init(self, wrapped)
self.s3fs = <CS3FileSystem*> wrapped.get()
- @classmethod
- def _reconstruct(cls, kwargs):
- return cls(**kwargs)
+ def _reconstruct(kwargs):
Review Comment:
TODO: I'm missing the @staticmethod keyword here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296196774
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Which docs? Cython docs?
It works fine in pure Python, but it may be a Cython wart, in which case it would be nice to report an issue:
```python
>>> class C:
...: @staticmethod
...: def sm(): pass
...:
>>> pickle.dumps(C.sm)
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x04C.sm\x94\x93\x94.'
>>> pickle.loads(pickle.dumps(C.sm))
<function __main__.C.sm()>
>>> pickle.loads(pickle.dumps(C().sm))
<function __main__.C.sm()>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729135198
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1678151878
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1721589828
I've separated out the enablement of Cython 3 into a separate diff: https://github.com/apache/arrow/pull/37743
Now, we can merge Cython 3 support and then enable Cython 3 once Cython 3.0.3 is released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` in cython 3, while in they are of type `enum` in cython 2.
Cython 3:
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> from enum import Enum
>>> class Color(Enum):
... red = 1
... blue = 2
...
>>> type(Color)
<class 'enum.EnumType'>
>>> Color
<enum 'Color'>
>>> Color.blue
<Color.blue: 2>
>>> type(Color.blue)
<enum 'Color'>
```
Cython 2:
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<enum 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
>>> type(pa.MetadataVersion.V5)
<enum 'MetadataVersion'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1308770376
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Ah bummer! I'll try fixing locally first. I was having trouble reproducing some of these last few warnings locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1310502578
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
> Can't we somehow indicate to numpydoc to skip those from the command line? I wouldn't put too much effort in this ..
Hmm, that might need some changes in the archery numpydoc integration .. (you can currently select a subset / module to test, but not exclude parts)
Looking at the numpydoc documentation, they indicate that the ignore commands also only work for "the CLI, pre-commit hook, and Sphinx extension" (https://numpydoc.readthedocs.io/en/latest/validation.html#ignoring-validation-checks-with-inline-comments), while I think archery uses their python API directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1310483600
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
BTW, on the latest commit, `FileSystemDataset.from_paths` isn't erroring anymore, so maybe the change from classmethod to staticmethod "fixed" it ..
Can't we somehow indicate to numpydoc to skip those from the command line? I wouldn't put too much effort in this ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: WIP: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1672105052
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296191120
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -439,6 +438,28 @@ def allow_append_to_file(request, filesystem_config):
return filesystem_config['allow_append_to_file']
+@pytest.fixture(
+ params=[
+ pytest.lazy_fixture('builtin_pickle'),
+ pytest.lazy_fixture('cloudpickle')
+ ]
+)
+def pickle(request):
+ return request.param
Review Comment:
Will do! I was thinking the same, but thought it might be better in a different PR. Now that you explicitly mention it, it is probably best to test cloudpickle everywhere as part of this cython upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296229950
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Yep, it was Cython docs. See section "Binding functions" here: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html
I'll reproduce the issue and share it here, and then file a cython issue if needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684077102
Revision: 2c1f54a2f7b08ca5720b4e636f4e54022e991fcd
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e17fa4f5f0](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e17fa4f5f0)
|Task|Status|
|----|------|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/5904331057/job/16016168240)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684073654
@github-actions crossbow submit test-conda-python-3.11-hypothesis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686912754
Revision: 5b1b75b4b7bb313fe9e9c7937f0dcd401b931338
Submitted crossbow builds: [ursacomputing/crossbow @ actions-19ec33868b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-19ec33868b)
|Task|Status|
|----|------|
|test-conda-python-3.11-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/5930598509/job/16080660018)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1684101482
@github-actions crossbow submit test-conda-python-3.9-pandas-latest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1696445416
Revision: 1e4a581347d3247c9795a967df9dca2dd2e03240
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1edb444ab9](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1edb444ab9)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/tree/actions-1edb444ab9-github-example-python-minimal-build-fedora-conda)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/6005194270/job/16287406410)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/6005180260/job/16287365779)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6005187672/job/16287386235)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6005196798/job/16287415170)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6005182876/job/16287373485)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6005187115/job/16287384678)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6005195060/job/16287409071)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6005180035/job/16287365156)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6005192761/job/16287401812)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6005183351/job/16287375099)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6005194706/job/16287407790)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6005186567/job/16287383254)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6005180825/job/16287367651)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6005193896/job/16287405068)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6005185771/job/16287381558)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6005184442/job/16287378329)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6005192220/job/16287400276)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6005197377/job/16287416864)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6005184959/job/16287379514)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6005191651/job/16287398602)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6005196270/job/16287413495)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6005193385/job/16287403447)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16287389321)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16287396991)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16287370905)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6005181937/job/16287370754)|
|verify-rc-source-python-linux-almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005183905/job/16287376510)|
|verify-rc-source-python-linux-conda-latest-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005188856/job/16287389533)|
|verify-rc-source-python-linux-ubuntu-20.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005191182/job/16287396955)|
|verify-rc-source-python-linux-ubuntu-22.04-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005190231/job/16287393805)|
|verify-rc-source-python-macos-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005195868/job/16287411910)|
|verify-rc-source-python-macos-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005182407/job/16287372109)|
|verify-rc-source-python-macos-conda-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/6005195489/job/16287410650)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309438420
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
I don't think the above is relevant. There just seem to be different types of python objects that get created in cython 2 vs 3 and some of them get parsed differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1699779500
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1327580461
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
FYI I haven't attempted #37235. I hope its okay to merge as-is given how long this PR has taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1328754248
##########
python/pyarrow/_substrait.pyx:
##########
@@ -28,7 +28,7 @@ from pyarrow.includes.libarrow_substrait cimport *
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
-):
+) noexcept:
Review Comment:
Will do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729169941
> We need to fix #37803 to use `@github-actions crossbow submit ...`. :<
Theaks! Got it now =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729164634
We need to fix #37803 to use `@github-actions crossbow submit ...`. :<
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1730433646
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit e83c23bb86156ef32d073c8eb507d601847350e1.
There was 1 benchmark result indicating a performance regression:
- Commit Run on `ursa-i9-9960x` at [2023-09-21 15:41:12Z](https://conbench.ursa.dev/compare/runs/12cb3f5ebd214645879db216d64795ed...a7c51cfb7494493eb11366ca144f90aa/)
- [`dataframe-to-table` (R) with dataset=type_floats, language=R](https://conbench.ursa.dev/compare/benchmarks/0650c489f755789c8000a730a23425a2...0650c7531843762b80007352a1ccc1bd)
The [full Conbench report](https://github.com/apache/arrow/runs/17021556517) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1687087200
Revision: 4c09badfdd688005f9671a8f0d3c9e5a18049d7c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5f16f1a67b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5f16f1a67b)
|Task|Status|
|----|------|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/5931767190/job/16084295495)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1307613229
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -771,13 +771,23 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
other.thrift_container_size_limit)
return attrs == other_attrs
+<<<<<<< HEAD
+<<<<<<< HEAD
@staticmethod
@binding(True) # Required for Cython < 3
+=======
+ @staticmethod
+>>>>>>> 12e6eb47d (Make _reconstruct staticmethods)
Review Comment:
```suggestion
```
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -771,13 +771,23 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
other.thrift_container_size_limit)
return attrs == other_attrs
+<<<<<<< HEAD
+<<<<<<< HEAD
Review Comment:
```suggestion
```
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -771,13 +771,23 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
other.thrift_container_size_limit)
return attrs == other_attrs
+<<<<<<< HEAD
+<<<<<<< HEAD
@staticmethod
@binding(True) # Required for Cython < 3
+=======
+ @staticmethod
+>>>>>>> 12e6eb47d (Make _reconstruct staticmethods)
def _reconstruct(kwargs):
# __reduce__ doesn't allow passing named arguments directly to the
# reconstructor, hence this wrapper.
return ParquetFragmentScanOptions(**kwargs)
+<<<<<<< HEAD
+=======
+>>>>>>> 0930ea44c (Fix cloudpickle test)
+=======
+>>>>>>> 12e6eb47d (Make _reconstruct staticmethods)
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1697352277
> `test-cuda-python` and `test-ubuntu-20.04-python-3` are failing due to this cython issue [cython/cython#5552](https://github.com/cython/cython/issues/5552)
Those are still failing. What do we do about those? Can we pin cython to cython<3 on those builds for now? Or disable gdb support on those builds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1309368053
##########
python/pyarrow/includes/libarrow_fs.pxd:
##########
@@ -23,7 +23,7 @@ from pyarrow.includes.libarrow_python cimport CTimePoint
cdef extern from "arrow/filesystem/api.h" namespace "arrow::fs" nogil:
- ctypedef enum CFileType "arrow::fs::FileType":
+ ctypedef enum CFileType "arrow::fs::FileType": # numpydoc ignore=PR01
Review Comment:
Hmm, the `cpdef enum` objects are all of type `flag` aka `enum.EnumType`.
```
...
<flag 'S3LogLevel'>
3
6
0
4
1
2
5
<class 'pyarrow._s3fs.S3FileSystem'>
<attribute 'region' of 'pyarrow._s3fs.S3FileSystem' objects>
<class 'pyarrow._fs.PyFileSystem'>
<attribute 'handler' of 'pyarrow._fs.PyFileSystem' objects>
<class 'pyarrow._fs.LocalFileSystem'>
<class 'pyarrow._hdfs.HadoopFileSystem'>
<cyfunction HadoopFileSystem.from_uri at 0x114c41b10>
<class 'pyarrow._gcsfs.GcsFileSystem'>
<attribute 'project_id' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<attribute 'default_bucket_location' of 'pyarrow._gcsfs.GcsFileSystem' objects>
<flag 'FileType'>
<class 'pyarrow._fs.FileSystemHandler'>
<cyfunction FileSystemHandler.open_output_stream at 0x114c7cba0>
...
```
```
>>> import pyarrow as pa
>>> pa.MetadataVersion
<flag 'MetadataVersion'>
>>> type(pa.MetadataVersion)
<class 'enum.EnumType'>
>>> type(pa.MetadataVersion.V5)
<flag 'MetadataVersion'>
>>> pa.MetadataVersion.V5
<MetadataVersion.V5: 4>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729156263
```
parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259489413
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF merged pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF merged PR #37097:
URL: https://github.com/apache/arrow/pull/37097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1729137812
```
parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259391389
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1328753329
##########
python/pyarrow/_dataset.pyx:
##########
@@ -1075,10 +1075,11 @@ cdef class FileSystemDataset(Dataset):
self.partition_expression
)
- @classmethod
- def from_paths(cls, paths, schema=None, format=None,
- filesystem=None, partitions=None, root_partition=None):
- """A Dataset created from a list of paths on a particular filesystem.
+ @staticmethod
+ def from_paths(paths, schema=None, format=None, filesystem=None,
+ partitions=None, root_partition=None):
Review Comment:
classmethod does work, I can change it back. I tried switching to staticmethod when numpydocs were failing, but it turns out it was the comment that was causing numpydoc parsing errors. I thought staticmethod was a slight improvement since `cls` wasn't actually used in the classmethod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1716015908
Revision: a10540f9c9a2f65881fdc315cc827893589452dd
Submitted crossbow builds: [ursacomputing/crossbow @ actions-40f522f834](https://github.com/ursacomputing/crossbow/branches/all?query=actions-40f522f834)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6162009845/job/16722539542)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6162015130/job/16722555177)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6162004226/job/16722522067)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6162008259/job/16722535460)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6162010694/job/16722541490)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6162007714/job/16722533733)|
|test-conda-python-3.10-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6162003013/job/16722518052)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6162016093/job/16722558683)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6162006027/job/16722528365)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6162007167/job/16722532362)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6162006541/job/16722529965)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6162000469/job/16722511384)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6162003678/job/16722520108)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6162011415/job/16722543537)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6162004879/job/16722524272)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6162005417/job/16722525947)|
|test-conda-python-3.8-spark-v3.4.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6162012322/job/16722545958)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6162001368/job/16722513499)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6162009209/job/16722537872)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6162002097/job/16722515530)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/16722514510)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/16722558787)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/16722553036)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6162014588/job/16722553083)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1721576204
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1701313987
@github-actions crossbow submit *python*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1295113131
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
+ return ParquetFragmentScanOptions(**kwargs)
Review Comment:
Can we check that this change still works with Cython < 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1298761676
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
Ahh I remember what happened. This works for Cython 3, but not Cython 0.29.36.
With Cython 0.29.36, we just need to add the `binding=True` compiler option.
my_test.pyx
```
# cython : language_level = 3
import cython
cdef class Foo():
@staticmethod
@cython.binding(True) # This works
def _reconstruct():
return Foo()
def __reduce__(self):
return Foo._reconstruct, tuple()
cdef class Bar():
@staticmethod # This does not work
def _reconstruct():
return Bar()
def __reduce__(self):
return Bar._reconstruct, tuple()
```
Usage here:
```
In [1]: from my_test import Foo, Bar
In [2]: f = Foo()
In [3]: b = Bar()
In [4]: import pickle
In [5]: pickle.dumps(f)
Out[5]: b'\x80\x04\x95#\x00\x00\x00\x00\x00\x00\x00\x8c\x07my_test\x94\x8c\x10Foo._reconstruct\x94\x93\x94)R\x94.'
In [6]: pickle.dumps(b)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call last)
Cell In[6], line 1
----> 1 pickle.dumps(b)
PicklingError: Can't pickle <built-in function _reconstruct>: it's not the same object as my_test._reconstruct
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1298788550
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -439,6 +438,28 @@ def allow_append_to_file(request, filesystem_config):
return filesystem_config['allow_append_to_file']
+@pytest.fixture(
+ params=[
+ pytest.lazy_fixture('builtin_pickle'),
+ pytest.lazy_fixture('cloudpickle')
+ ]
+)
+def pickle(request):
+ return request.param
Review Comment:
The commit to implement this is pretty big (I already pushed it to this PR). I went ahead and implemented in a separate PR and can rebase on top of it after it merges. This has the additional benefit of testing the CI process with Cython 0.29.X, since I haven't added the Cython < 3 CI job yet.
https://github.com/apache/arrow/pull/37255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on a diff in pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #37097:
URL: https://github.com/apache/arrow/pull/37097#discussion_r1296183189
##########
python/pyarrow/_dataset_parquet.pyx:
##########
@@ -741,7 +737,15 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
thrift_string_size_limit=self.thrift_string_size_limit,
thrift_container_size_limit=self.thrift_container_size_limit,
)
- return type(self)._reconstruct, (kwargs,)
+ return _reconstruct_parquet_fragment_scan_options, (kwargs,)
+
+
+def _reconstruct_parquet_fragment_scan_options(kwargs):
+ # __reduce__ doesn't allow passing named arguments directly to the
+ # reconstructor, hence this wrapper.
+ # In Cython >= 3.0.0, function binding is turned on by default, so
+ # a global static method is used (instead of a class method) for pickling.
Review Comment:
`_reconstruct` as a staticmethod failed with both builtin pickle and cloudpickle, but I can't remember the exact error. I can grab it for you if you'd like. You're right that we should also be able to add a cython decorator to disable binding if that's preferred. I'll go ahead and update with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1681304913
> > Numpydocs now correctly reports 72 cython functions are missing documentation for their parameters (Should be fixed in a separate PR IMO)
>
> Could you work on this before we merge this to avoid CI failures on main?
Created https://github.com/apache/arrow/pull/37218 ! This fixes most of the issues. I figured we can merge that, rebase here, and then fix anything leftover, which will be manageable to do in this PR. We won't be able to verify the numpydocs test in that PR because it doesn't have cython 3. I tested it with cython 3 locally instead, but didn't see the exact same number of issues that are reported here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] danepitkin commented on pull request #37097: GH-36730: [Python] Add support for Cython 3.0.0
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on PR #37097:
URL: https://github.com/apache/arrow/pull/37097#issuecomment-1686894215
@github actions crossbow submit test-conda-python-3.11-cython2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org