You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2025/12/22 19:53:07 UTC
[PR] GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
rok opened a new pull request, #48622:
URL: https://github.com/apache/arrow/pull/48622
### Rationale for this change
This is the second in series of PRs adding type annotations to pyarrow and resolving #32609. It builds on top of and should be merged after #48618.
### What changes are included in this PR?
This adds:
- `_types.pyi` - Core type definitions including
- `_stubs_typing.pyi` - Internal typing protocols and helpers used across stub files
- `error.pyi` - Exception classes (`ArrowException`, `ArrowInvalid`, `ArrowIOError`, etc.)
- Minimal placeholder stubs - `lib.pyi`, `io.pyi`, `scalar.pyi` - using `__getattr__` to allow imports to resolve while deferring to subsequent PRs
### Are these changes tested?
Via CI type checks established in #48618.
### Are there any user-facing changes?
Users will start seeing some minimal annotated types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4175876636
> The error I was getting without the CMake change was:
>
> ```python
> >>> import pyarrow
> Traceback (most recent call last):
> File "<python-input-0>", line 1, in <module>
> import pyarrow
> File "/Users/rok/Documents/repos/arrow/python/pyarrow/__init__.py", line 59, in <module>
> from pyarrow.lib import (BuildInfo, CppBuildInfo, RuntimeInfo, set_timezone_db_path,
> ...<3 lines>...
> io_thread_count, is_opentelemetry_enabled, set_io_thread_count)
> ModuleNotFoundError: No module named 'pyarrow.lib'
> ```
>
> With the CMake change it works fine.
>
> In both cases I'm doing:
>
> ```shell
> export ARROW_HOME=$(pwd)/dist
> export CMAKE_PREFIX_PATH=$ARROW_HOME:$CMAKE_PREFIX_PATH
> export DYLD_LIBRARY_PATH=$ARROW_HOME/lib:$DYLD_LIBRARY_PATH
> uv pip install --no-build-isolation --editable .
> ```
@rok I think this would now be fixed with rebase, see: https://github.com/apache/arrow/issues/49566.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2936909497
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportEq(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+class SupportLt(Protocol):
+ def __lt__(self, other: object, /) -> bool: ...
+
+class SupportGt(Protocol):
+ def __gt__(self, other: object, /) -> bool: ...
+
+class SupportLe(Protocol):
+ def __le__(self, other: object, /) -> bool: ...
+
+class SupportGe(Protocol):
+ def __ge__(self, other: object, /) -> bool: ...
+
+FilterTuple: TypeAlias = (
+ tuple[str, Literal["=", "==", "!="], SupportEq]
+ | tuple[str, Literal["<"], SupportLt]
+ | tuple[str, Literal[">"], SupportGt]
+ | tuple[str, Literal["<="], SupportLe]
+ | tuple[str, Literal[">="], SupportGe]
+ | tuple[str, Literal["in", "not in"], Collection]
+ | tuple[str, str, Any] # Allow general str for operator to avoid type errors
+)
+
+class Buffer(Protocol): ...
+class SupportPyBuffer(Protocol): ...
+
+class SupportArrowStream(Protocol):
+ def __arrow_c_stream__(self, requested_schema=None, /) -> Any: ...
+
+class SupportPyArrowArray(Protocol):
+ def __arrow_array__(self, type=None, /) -> Any: ...
+
+class SupportArrowArray(Protocol):
+ def __arrow_c_array__(self, requested_schema=None, /) -> Any: ...
+
+class SupportArrowDeviceArray(Protocol):
+ def __arrow_c_device_array__(self, requested_schema=None, /, **kwargs) -> Any: ...
+
+class SupportArrowSchema(Protocol):
+ def __arrow_c_schema__(self) -> Any: ...
Review Comment:
Very minor nit, but the [naming convention](https://typing.python.org/en/latest/guides/writing_stubs.html#naming) for these is normally:
- ~~`SupportX`~~
- `SupportsX`
There's some variation on this that are shorter in (https://numpy.org/numtype/dev/style/#protocol-names) too
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
Review Comment:
I do get the pragmatic reason for adding these *generally* - but `list_size` is a strange one.
Only `FixedSizeListType.list_size` exists, is it more commonly accessed than the other `*ListType` properties?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
Review Comment:
```suggestion
def field(self, i: int) -> Field[Any]: ...
```
You can get these flagged with [`reportMissingTypeArgument`](https://github.com/microsoft/pylance-release/blob/f1b9ceea31792f6acd2fc2ef77654e4418d5033c/docs/diagnostics/reportMissingTypeArgument.md) or [`disallow-any-generics`](https://mypy.readthedocs.io/en/stable/error_code_list2.html#check-that-type-arguments-exist-type-arg)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
Review Comment:
You could probably do something similar for `*ListType`s too
```suggestion
from typing import Protocol
_Precision = TypeVar("_Precision", default=Any , covariant=True)
_Scale = TypeVar("_Scale", default=Any , covariant=True)
class _HasPrecisionScale(Protocol[_Precision, _Scale]):
@property
def precision(self) -> _Precision: ...
@property
def scale(self) -> _Scale: ...
class Decimal32Type(FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision, _Scale]): ...
class Decimal64Type(FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision, _Scale]): ...
class Decimal128Type(FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision, _Scale]): ...
class Decimal256Type(FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision, _Scale]): ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
+def sparse_union(
+ child_fields: list[Field[Any]], type_codes: list[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: list[Field[Any]], type_codes: list[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: list[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: list[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | str | None, value_type: _BasicValueT | str | None
+) -> RunEndEncodedType[_RunEndType, _BasicValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
+ permutation: Sequence[int] | None = None,
+) -> FixedShapeTensorType[_ValueT]: ...
+def bool8() -> Bool8Type: ...
+def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
+def type_for_alias(name: Any) -> DataType: ...
+def schema(
+ fields: (
+ Iterable[Field[Any]]
+ | Iterable[tuple[str, DataType | str | None]]
+ | Mapping[Any, DataType | str | None]
+ ),
+ metadata: Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+ | None = None,
Review Comment:
Seems like a good candidate for a `TypeAlias` maybe?
It's annoying that you can't just write `Mapping[str | bytes, str | bytes]` (https://github.com/python/typing/pull/273)
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportEq(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+class SupportLt(Protocol):
+ def __lt__(self, other: object, /) -> bool: ...
+
+class SupportGt(Protocol):
+ def __gt__(self, other: object, /) -> bool: ...
+
+class SupportLe(Protocol):
+ def __le__(self, other: object, /) -> bool: ...
+
+class SupportGe(Protocol):
+ def __ge__(self, other: object, /) -> bool: ...
+
+FilterTuple: TypeAlias = (
+ tuple[str, Literal["=", "==", "!="], SupportEq]
+ | tuple[str, Literal["<"], SupportLt]
+ | tuple[str, Literal[">"], SupportGt]
+ | tuple[str, Literal["<="], SupportLe]
+ | tuple[str, Literal[">="], SupportGe]
+ | tuple[str, Literal["in", "not in"], Collection]
+ | tuple[str, str, Any] # Allow general str for operator to avoid type errors
+)
+
+class Buffer(Protocol): ...
+class SupportPyBuffer(Protocol): ...
Review Comment:
Is there a plan for these to be extended later?
I imagine using a `Protocol` with no members is equivalent to `object`
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportEq(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+class SupportLt(Protocol):
+ def __lt__(self, other: object, /) -> bool: ...
+
+class SupportGt(Protocol):
+ def __gt__(self, other: object, /) -> bool: ...
+
+class SupportLe(Protocol):
+ def __le__(self, other: object, /) -> bool: ...
+
+class SupportGe(Protocol):
+ def __ge__(self, other: object, /) -> bool: ...
Review Comment:
You might be able to reuse the (https://github.com/python/typeshed) versions of these
- https://github.com/python/typeshed/blob/466f4f2744767d6356f7e9f398ba33587b620214/stdlib/_typeshed/__init__.pyi#L89-L100
AFAIK, type checkers should let you do imports like this:
```py
from _typeshed import SupportsDunderLT
```
And it would be referencing the [version they have vendored](https://github.com/python/typeshed#using) - instead of working like a normal dependency
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportEq(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+class SupportLt(Protocol):
+ def __lt__(self, other: object, /) -> bool: ...
+
+class SupportGt(Protocol):
+ def __gt__(self, other: object, /) -> bool: ...
+
+class SupportLe(Protocol):
+ def __le__(self, other: object, /) -> bool: ...
+
+class SupportGe(Protocol):
+ def __ge__(self, other: object, /) -> bool: ...
+
+FilterTuple: TypeAlias = (
+ tuple[str, Literal["=", "==", "!="], SupportEq]
+ | tuple[str, Literal["<"], SupportLt]
+ | tuple[str, Literal[">"], SupportGt]
+ | tuple[str, Literal["<="], SupportLe]
+ | tuple[str, Literal[">="], SupportGe]
+ | tuple[str, Literal["in", "not in"], Collection]
+ | tuple[str, str, Any] # Allow general str for operator to avoid type errors
+)
+
+class Buffer(Protocol): ...
+class SupportPyBuffer(Protocol): ...
+
+class SupportArrowStream(Protocol):
+ def __arrow_c_stream__(self, requested_schema=None, /) -> Any: ...
+
+class SupportPyArrowArray(Protocol):
+ def __arrow_array__(self, type=None, /) -> Any: ...
+
+class SupportArrowArray(Protocol):
+ def __arrow_c_array__(self, requested_schema=None, /) -> Any: ...
+
+class SupportArrowDeviceArray(Protocol):
+ def __arrow_c_device_array__(self, requested_schema=None, /, **kwargs) -> Any: ...
+
+class SupportArrowSchema(Protocol):
+ def __arrow_c_schema__(self) -> Any: ...
+
+class NullableCollection(Protocol[_V]):
+ def __iter__(self) -> Iterator[_V] | Iterator[_V | None]: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, item: Any, /) -> bool: ...
Review Comment:
Well this is interesting...
So this came from [zen-xu/pyarrow-stubs diff](https://github.com/zen-xu/pyarrow-stubs/commit/213230fe6aed839191d75724ff34d44d133a4bc7#diff-3c8607d9365eff87d49b38bb1669c985d4ddb00437a79b1379789fda929eef58)
I think the original issue was that this type was problematic:
```py
CollectionValue: TypeAlias = list[_V | None] | tuple[_V | None, ...] | set[_V | None]
```
It looks like you fixed the variance issue `pyarrow-stubs` has, nice one 😄
https://github.com/zen-xu/pyarrow-stubs/blob/148368c532f96f953cf39f34120da9d1043bb4dc/pyarrow-stubs/__lib_pxi/scalar.pyi#L390-L395
<details><summary>All of these check out</summary>
<p>
```py
a: NullableCollection[int] = []
b: NullableCollection[int] = [1]
c: NullableCollection[int] = [1, None]
d: NullableCollection[int] = [None]
e: NullableCollection[int] = ()
f: NullableCollection[int] = (1,)
g: NullableCollection[int] = (1, None)
h: NullableCollection[int] = (None,)
```
</p>
</details>
---
May I offer this slight tweak?
Has the same meaning - but the protocol is defined by how it *isn't* the same as [`Collection`](https://github.com/python/typeshed/blob/466f4f2744767d6356f7e9f398ba33587b620214/stdlib/typing.pyi#L651-L656).
Which has [this `__iter__`](https://github.com/python/typeshed/blob/466f4f2744767d6356f7e9f398ba33587b620214/stdlib/typing.pyi#L529)
```suggestion
from collections.abc import Container, Sized
class NullableCollection(Sized, Container[_V], Protocol[_V]):
def __iter__(self) -> Iterator[_V] | Iterator[_V | None]: ...
```
##########
.pre-commit-config.yaml:
##########
@@ -136,11 +136,22 @@ repos:
- id: flake8
alias: python
name: Python Lint
+ additional_dependencies:
Review Comment:
(Just for reference)
This is what our `pre-commit` looks like, since we have `ruff` and `flake8` in there - the answer to (https://github.com/apache/arrow/pull/48622#issuecomment-4056395225) is probably something similar 🙂
- https://github.com/narwhals-dev/narwhals/blob/061c97f8a01bf9e721835978b039303c5051501c/.pre-commit-config.yaml#L4-L16
- https://github.com/narwhals-dev/narwhals/blob/061c97f8a01bf9e721835978b039303c5051501c/.pre-commit-config.yaml#L29-L57
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
Review Comment:
```suggestion
_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
_DataTypeT_co = TypeVar("_DataTypeT", bound=DataType, covariant=True)
```
The `covariant` version should be used on all the classes that currently use `_DataTypeT` **if either**:
1. `_DataTypeT` only appears as a **return type** (including `@property`) [^1]
2. `_DataTypeT` *also* appears as a parameter, but exclusively in `__init__` [^2]
I think with `pyarrow` generally a lot of types will be [covariant](https://typing.python.org/en/latest/spec/generics.html#variance), since you can't (easily?) mutate things from python
[^1]: If you write the classes entirely as `Protocol`s, a type checker tells you when you [*could* use covariance](https://typing.python.org/en/latest/spec/protocol.html#generic-protocols).
Not sure why they don't do it for regular classes too 😢
[^2]: The exception for `__init__` is annoying for the `from_*` constructor pattern (which I also use a lot)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
Review Comment:
This would be another spot for the `schema(, metadata)` `TypeAlias` IIUC?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
Review Comment:
I think this is hiding some ergonomics that the runtime version has:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5335-L5343
So the what gets accepted by `field` would be defined by:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5761-L5852
--
## Suggestion
1. add those aliases as a `TypeAlias`
2. add another one which is `DataType | <1> | None`
3. Use that in `field(type: <2>)`
4. Then add update `struct` (and any other places) to use the same when they share the dependency 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4054882255
> @dangotbanned with packaging part mostly solved I'd like to get this and maybe #49200 through review this week.
> Any chance you'd have time to review? :)
Thanks for the ping @rok
Yeah I will try my best to review this week.
Couple quick questions for now:
1. Are you just looking for a review of:
- [`_stubs_typing.pyi`]
- [`_types.pyi`]
- [`error.pyi`]
2. Have you considered using [Ruff] to help with the stubs?
- There is a ([`PYI`]) group that mostly overlaps with the [Writing and Maintaining Stub Files] guide.
- IIRC, it only applies to `.pyi` files, but Ruff is pretty easy to configure for a specific directory anyway 🙂
- You may also be interested in running the formatter on `.pyi` files, as they could be a lot denser.
- I've linked an example of what `_types.pyi` looks like after dropping it into `narwhals` (963 -> 654 LOC)
- Here's a couple more from `numpy` ([1], [2], [3]), which uses [this config]
<details><summary><b>Show ruff-er <code>_types.pyi</code></b></summary>
<p>
Source: (https://github.com/rok/arrow/blob/a68ed19a650e4069076262642cb9ad91cad1a100/python/pyarrow-stubs/pyarrow/_types.pyi)
Config: (https://github.com/narwhals-dev/narwhals/blob/061c97f8a01bf9e721835978b039303c5051501c/pyproject.toml#L137-L246)
```pyi
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import datetime as dt
from collections.abc import Iterable, Iterator, Mapping, Sequence
from decimal import Decimal
from typing import Any, Generic, Literal
import numpy as np
import pandas as pd
from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
from pyarrow.io import Buffer
from pyarrow.lib import (
Array,
ChunkedArray,
ExtensionArray,
MemoryPool,
MonthDayNano,
Table,
)
from pyarrow.scalar import ExtensionScalar
from typing_extensions import Self, TypeVar, deprecated
class _Weakrefable: ...
class _Metadata(_Weakrefable): ...
class DataType(_Weakrefable):
def field(self, i: int) -> Field: ...
@property
def id(self) -> int: ...
@property
def bit_width(self) -> int: ...
@property
def byte_width(self) -> int: ...
@property
def num_fields(self) -> int: ...
@property
def num_buffers(self) -> int: ...
@property
def has_variadic_buffers(self) -> bool: ...
# Properties that exist on specific subtypes but accessed generically
@property
def list_size(self) -> int: ...
def __hash__(self) -> int: ...
def equals(self, other: DataType | str, *, check_metadata: bool = False) -> bool: ...
def to_pandas_dtype(self) -> np.generic: ...
def _export_to_c(self, out_ptr: int) -> None: ...
@classmethod
def _import_from_c(cls, in_ptr: int) -> Self: ...
def __arrow_c_schema__(self) -> Any: ...
@classmethod
def _import_from_c_capsule(cls, schema) -> Self: ...
_AsPyType = TypeVar("_AsPyType")
_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
class _BasicDataType(DataType, Generic[_AsPyType]): ...
class NullType(_BasicDataType[None]): ...
class BoolType(_BasicDataType[bool]): ...
class UInt8Type(_BasicDataType[int]): ...
class Int8Type(_BasicDataType[int]): ...
class UInt16Type(_BasicDataType[int]): ...
class Int16Type(_BasicDataType[int]): ...
class UInt32Type(_BasicDataType[int]): ...
class Int32Type(_BasicDataType[int]): ...
class UInt64Type(_BasicDataType[int]): ...
class Int64Type(_BasicDataType[int]): ...
class Float16Type(_BasicDataType[float]): ...
class Float32Type(_BasicDataType[float]): ...
class Float64Type(_BasicDataType[float]): ...
class Date32Type(_BasicDataType[dt.date]): ...
class Date64Type(_BasicDataType[dt.date]): ...
class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
class StringType(_BasicDataType[str]): ...
class LargeStringType(_BasicDataType[str]): ...
class StringViewType(_BasicDataType[str]): ...
class BinaryType(_BasicDataType[bytes]): ...
class LargeBinaryType(_BasicDataType[bytes]): ...
class BinaryViewType(_BasicDataType[bytes]): ...
_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
_Tz = TypeVar("_Tz", str, None, default=None)
class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
@property
def unit(self) -> _Unit: ...
@property
def tz(self) -> _Tz: ...
_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
@property
def unit(self) -> _Time32Unit: ...
_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
@property
def unit(self) -> _Time64Unit: ...
class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
@property
def unit(self) -> _Unit: ...
_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
class FixedSizeBinaryType(
_BasicDataType[_FixedSizeBinaryAsPyType], Generic[_FixedSizeBinaryAsPyType]
): ...
_Precision = TypeVar("_Precision", default=Any)
_Scale = TypeVar("_Scale", default=Any)
class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
@property
def precision(self) -> _Precision: ...
@property
def scale(self) -> _Scale: ...
class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
@property
def precision(self) -> _Precision: ...
@property
def scale(self) -> _Scale: ...
class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
@property
def precision(self) -> _Precision: ...
@property
def scale(self) -> _Scale: ...
class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
@property
def precision(self) -> _Precision: ...
@property
def scale(self) -> _Scale: ...
class ListType(DataType, Generic[_DataTypeT]):
@property
def value_field(self) -> Field[_DataTypeT]: ...
@property
def value_type(self) -> _DataTypeT: ...
class LargeListType(DataType, Generic[_DataTypeT]):
@property
def value_field(self) -> Field[_DataTypeT]: ...
@property
def value_type(self) -> _DataTypeT: ...
class ListViewType(DataType, Generic[_DataTypeT]):
@property
def value_field(self) -> Field[_DataTypeT]: ...
@property
def value_type(self) -> _DataTypeT: ...
class LargeListViewType(DataType, Generic[_DataTypeT]):
@property
def value_field(self) -> Field[_DataTypeT]: ...
@property
def value_type(self) -> _DataTypeT: ...
class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
@property
def value_field(self) -> Field[_DataTypeT]: ...
@property
def value_type(self) -> _DataTypeT: ...
@property
def list_size(self) -> int: ...
class DictionaryMemo(_Weakrefable): ...
_IndexT = TypeVar(
"_IndexT",
UInt8Type,
Int8Type,
UInt16Type,
Int16Type,
UInt32Type,
Int32Type,
UInt64Type,
Int64Type,
)
_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
_ValueT = TypeVar("_ValueT", bound=DataType)
_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
@property
def ordered(self) -> _Ordered: ...
@property
def index_type(self) -> _IndexT: ...
@property
def value_type(self) -> _BasicValueT: ...
_K = TypeVar("_K", bound=DataType)
class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
@property
def key_field(self) -> Field[_K]: ...
@property
def key_type(self) -> _K: ...
@property
def item_field(self) -> Field[_ValueT]: ...
@property
def item_type(self) -> _ValueT: ...
@property
def keys_sorted(self) -> _Ordered: ...
_Size = TypeVar("_Size", default=int)
class StructType(DataType):
def get_field_index(self, name: str) -> int: ...
def field(self, i: int | str) -> Field: ...
def get_all_field_indices(self, name: str) -> list[int]: ...
def __len__(self) -> int: ...
def __iter__(self) -> Iterator[Field]: ...
__getitem__ = field
@property
def names(self) -> list[str]: ...
@property
def fields(self) -> list[Field]: ...
class UnionType(DataType):
@property
def mode(self) -> Literal["sparse", "dense"]: ...
@property
def type_codes(self) -> list[int]: ...
def __len__(self) -> int: ...
def __iter__(self) -> Iterator[Field]: ...
def field(self, i: int) -> Field: ...
__getitem__ = field
class SparseUnionType(UnionType):
@property
def mode(self) -> Literal["sparse"]: ...
class DenseUnionType(UnionType):
@property
def mode(self) -> Literal["dense"]: ...
_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
@property
def run_end_type(self) -> _RunEndType: ...
@property
def value_type(self) -> _BasicValueT: ...
_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
class BaseExtensionType(DataType):
def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
@property
def extension_name(self) -> str: ...
@property
def storage_type(self) -> DataType: ...
def wrap_array(self, storage: _StorageT) -> _StorageT: ...
class ExtensionType(BaseExtensionType):
def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
def __arrow_ext_serialize__(self) -> bytes: ...
@classmethod
def __arrow_ext_deserialize__(
cls, storage_type: DataType, serialized: bytes
) -> Self: ...
class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
@property
def value_type(self) -> _ValueT: ...
@property
def shape(self) -> list[int]: ...
@property
def dim_names(self) -> list[str] | None: ...
@property
def permutation(self) -> list[int] | None: ...
class Bool8Type(BaseExtensionType): ...
class UuidType(BaseExtensionType): ...
class JsonType(BaseExtensionType): ...
class OpaqueType(BaseExtensionType):
@property
def type_name(self) -> str: ...
@property
def vendor_name(self) -> str: ...
class UnknownExtensionType(ExtensionType):
def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
def register_extension_type(ext_type: ExtensionType) -> None: ...
def unregister_extension_type(type_name: str) -> None: ...
class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
def __init__(
self,
__arg0__: Mapping[str | bytes, str | bytes]
| Iterable[tuple[str | bytes, str | bytes]]
| KeyValueMetadata
| None = None,
**kwargs: str,
) -> None: ...
def equals(self, other: KeyValueMetadata) -> bool: ...
def __len__(self) -> int: ...
def __contains__(self, /, __key: object) -> bool: ...
def __getitem__(self, /, __key: Any) -> Any: ...
def __iter__(self) -> Iterator[bytes]: ...
def get_all(self, key: str) -> list[bytes]: ...
def to_dict(self) -> dict[bytes, bytes]: ...
class Field(_Weakrefable, Generic[_DataTypeT]):
def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
def __hash__(self) -> int: ...
@property
def nullable(self) -> bool: ...
@property
def name(self) -> str: ...
@property
def metadata(self) -> dict[bytes, bytes] | None: ...
@property
def type(self) -> _DataTypeT: ...
def with_metadata(
self,
metadata: dict[bytes | str, bytes | str]
| Mapping[bytes | str, bytes | str]
| Any,
) -> Self: ...
def remove_metadata(self) -> Self: ...
def with_type(self, new_type: DataType) -> Field: ...
def with_name(self, name: str) -> Self: ...
def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
def flatten(self) -> list[Field]: ...
def _export_to_c(self, out_ptr: int) -> None: ...
@classmethod
def _import_from_c(cls, in_ptr: int) -> Self: ...
def __arrow_c_schema__(self) -> Any: ...
@classmethod
def _import_from_c_capsule(cls, schema) -> Self: ...
class Schema(_Weakrefable):
def __len__(self) -> int: ...
def __getitem__(self, key: str | int) -> Field: ...
_field = __getitem__
def __iter__(self) -> Iterator[Field]: ...
def __hash__(self) -> int: ...
def __sizeof__(self) -> int: ...
@property
def pandas_metadata(self) -> dict: ...
@property
def names(self) -> list[str]: ...
@property
def types(self) -> list[DataType]: ...
@property
def metadata(self) -> dict[bytes, bytes]: ...
def empty_table(self) -> Table: ...
def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
@classmethod
def from_pandas(
cls, df: pd.DataFrame, preserve_index: bool | None = None
) -> Schema: ...
def field(self, i: int | str | bytes) -> Field: ...
@deprecated("Use 'field' instead")
def field_by_name(self, name: str) -> Field: ...
def get_field_index(self, name: str) -> int: ...
def get_all_field_indices(self, name: str) -> list[int]: ...
def append(self, field: Field) -> Schema: ...
def insert(self, i: int, field: Field) -> Schema: ...
def remove(self, i: int) -> Schema: ...
def set(self, i: int, field: Field) -> Schema: ...
@deprecated("Use 'with_metadata' instead")
def add_metadata(self, metadata: dict) -> Schema: ...
def with_metadata(self, metadata: dict) -> Schema: ...
def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
def remove_metadata(self) -> Schema: ...
def to_string(
self,
truncate_metadata: bool = True,
show_field_metadata: bool = True,
show_schema_metadata: bool = True,
element_size_limit: int | None = None,
) -> str: ...
def _export_to_c(self, out_ptr: int) -> None: ...
@classmethod
def _import_from_c(cls, in_ptr: int) -> Schema: ...
def __arrow_c_schema__(self) -> Any: ...
@staticmethod
def _import_from_c_capsule(schema: Any) -> Schema: ...
def unify_schemas(
schemas: Sequence[Schema],
*,
promote_options: Literal["default", "permissive"] = "default",
) -> Schema: ...
def field(
name: SupportArrowSchema | str | Any,
type: _DataTypeT | str | None = None,
nullable: bool = True,
metadata: dict[Any, Any] | None = None,
) -> Field[_DataTypeT] | Field[Any]: ...
def null() -> NullType: ...
def bool_() -> BoolType: ...
def uint8() -> UInt8Type: ...
def int8() -> Int8Type: ...
def uint16() -> UInt16Type: ...
def int16() -> Int16Type: ...
def uint32() -> UInt32Type: ...
def int32() -> Int32Type: ...
def int64() -> Int64Type: ...
def uint64() -> UInt64Type: ...
def timestamp(unit: _Unit | str, tz: _Tz | None = None) -> TimestampType[_Unit, _Tz]: ...
def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
def date32() -> Date32Type: ...
def date64() -> Date64Type: ...
def float16() -> Float16Type: ...
def float32() -> Float32Type: ...
def float64() -> Float64Type: ...
def decimal32(
precision: _Precision, scale: _Scale | Literal[0] = 0
) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
def decimal64(
precision: _Precision, scale: _Scale | Literal[0] = 0
) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
def decimal128(
precision: _Precision, scale: _Scale | Literal[0] = 0
) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
def decimal256(
precision: _Precision, scale: _Scale | Literal[0] = 0
) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
def string() -> StringType: ...
utf8 = string
def binary(
length: Literal[-1] | int = ...,
) -> BinaryType | FixedSizeBinaryType[bytes]: ...
def large_binary() -> LargeBinaryType: ...
def large_string() -> LargeStringType: ...
large_utf8 = large_string
def binary_view() -> BinaryViewType: ...
def string_view() -> StringViewType: ...
def list_(
value_type: _DataTypeT | Field[_DataTypeT] | None = None,
list_size: Literal[-1] | _Size | None = None,
) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
def large_list(
value_type: _DataTypeT | Field[_DataTypeT] | None = None,
) -> LargeListType[_DataTypeT]: ...
def list_view(
value_type: _DataTypeT | Field[_DataTypeT] | None = None,
) -> ListViewType[_DataTypeT]: ...
def large_list_view(
value_type: _DataTypeT | Field[_DataTypeT] | None = None,
) -> LargeListViewType[_DataTypeT]: ...
def map_(
key_type: _K | Field | str | None = None,
item_type: _ValueT | Field | str | None = None,
keys_sorted: _Ordered | None = None,
) -> MapType[_K, _ValueT, _Ordered]: ...
def dictionary(
index_type: _IndexT | str,
value_type: _BasicValueT | str,
ordered: _Ordered | None = None,
) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
def struct(
fields: Iterable[
Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
]
| Mapping[str, Field[Any] | DataType | None],
) -> StructType: ...
def sparse_union(
child_fields: list[Field[Any]], type_codes: list[int] | None = None
) -> SparseUnionType: ...
def dense_union(
child_fields: list[Field[Any]], type_codes: list[int] | None = None
) -> DenseUnionType: ...
def union(
child_fields: list[Field[Any]],
mode: Literal["sparse", "dense"] | int | str,
type_codes: list[int] | None = None,
) -> SparseUnionType | DenseUnionType: ...
def run_end_encoded(
run_end_type: _RunEndType | str | None, value_type: _BasicValueT | str | None
) -> RunEndEncodedType[_RunEndType, _BasicValueT]: ...
def json_(storage_type: DataType = ...) -> JsonType: ...
def uuid() -> UuidType: ...
def fixed_shape_tensor(
value_type: _ValueT,
shape: Sequence[int],
dim_names: Sequence[str] | None = None,
permutation: Sequence[int] | None = None,
) -> FixedShapeTensorType[_ValueT]: ...
def bool8() -> Bool8Type: ...
def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
def type_for_alias(name: Any) -> DataType: ...
def schema(
fields: (
Iterable[Field[Any]]
| Iterable[tuple[str, DataType | str | None]]
| Mapping[Any, DataType | str | None]
),
metadata: Mapping[bytes, bytes]
| Mapping[str, str]
| Mapping[bytes, str]
| Mapping[str, bytes]
| None = None,
) -> Schema: ...
def from_numpy_dtype(dtype: np.dtype[Any] | type | str) -> DataType: ...
__all__ = [
"BaseExtensionType",
"BinaryType",
"BinaryViewType",
"Bool8Type",
"BoolType",
"DataType",
"Date32Type",
"Date64Type",
"Decimal32Type",
"Decimal64Type",
"Decimal128Type",
"Decimal256Type",
"DenseUnionType",
"DictionaryMemo",
"DictionaryType",
"DurationType",
"ExtensionType",
"Field",
"FixedShapeTensorType",
"FixedSizeBinaryType",
"FixedSizeListType",
"Float16Type",
"Float32Type",
"Float64Type",
"Int8Type",
"Int16Type",
"Int32Type",
"Int64Type",
"JsonType",
"KeyValueMetadata",
"LargeBinaryType",
"LargeListType",
"LargeListViewType",
"LargeStringType",
"ListType",
"ListViewType",
"MapType",
"MonthDayNanoIntervalType",
"NullType",
"OpaqueType",
"RunEndEncodedType",
"Schema",
"SparseUnionType",
"StringType",
"StringViewType",
"StructType",
"Time32Type",
"Time64Type",
"TimestampType",
"UInt8Type",
"UInt16Type",
"UInt32Type",
"UInt64Type",
"UnionType",
"UnknownExtensionType",
"UuidType",
"_BasicDataType",
"_DataTypeT",
"_Metadata",
"_Time32Unit",
"_Time64Unit",
"_Tz",
"_Unit",
"_Weakrefable",
"binary",
"binary_view",
"bool8",
"bool_",
"date32",
"date64",
"decimal32",
"decimal64",
"decimal128",
"decimal256",
"dense_union",
"dictionary",
"duration",
"field",
"fixed_shape_tensor",
"float16",
"float32",
"float64",
"from_numpy_dtype",
"int8",
"int16",
"int32",
"int64",
"json_",
"large_binary",
"large_list",
"large_list_view",
"large_string",
"large_utf8",
"list_",
"list_view",
"map_",
"month_day_nano_interval",
"null",
"opaque",
"register_extension_type",
"run_end_encoded",
"schema",
"sparse_union",
"string",
"string_view",
"struct",
"time32",
"time64",
"timestamp",
"type_for_alias",
"uint8",
"uint16",
"uint32",
"uint64",
"unify_schemas",
"union",
"unregister_extension_type",
"utf8",
"uuid",
]
```
</p>
</details>
[`_stubs_typing.pyi`]: https://github.com/apache/arrow/pull/48622/files#diff-c5785163a5f2f9960c4b1d90d2fdc8652e137433af5d88d8a3819a613c4a1b79
[`_types.pyi`]: https://github.com/apache/arrow/pull/48622/files#diff-9558daf1f32eda5aedc71762fbc38dc60a18634dfb05d7c77f8911e1d25537e0
[`error.pyi`]: https://github.com/apache/arrow/pull/48622/files#diff-83c40a21fb6ef9b823e58c402b7c67637dccfe382c49342ead56bdcb45424021
[Ruff]: https://docs.astral.sh/ruff/
[`PYI`]: https://docs.astral.sh/ruff/rules/#flake8-pyi-pyi
[Writing and Maintaining Stub Files]: https://typing.python.org/en/latest/guides/writing_stubs.html
[1]: https://github.com/numpy/numpy/blob/581d10f43b539a189a2d37856e5130464de9e5f6/numpy/_core/multiarray.pyi
[2]: https://github.com/numpy/numpy/blob/581d10f43b539a189a2d37856e5130464de9e5f6/numpy/__init__.pyi
[3]: https://github.com/numpy/numpy/blob/581d10f43b539a189a2d37856e5130464de9e5f6/numpy/dtypes.pyi
[this config]: https://github.com/numpy/numpy/blob/581d10f43b539a189a2d37856e5130464de9e5f6/ruff.toml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r4085902882
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-not-found,import-untyped]
Review Comment:
Hi @samwgoldman! I created https://github.com/apache/arrow/issues/51474 to track this. I don't think we need to wait for this to complete to add it. Also FYI we're expecting to be able to put some real effort into annotations from first week of October on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4056395225
@dangotbanned I ran ruff linter and formatter over the source. Looks much nicer now. I also added `flake8-pyi` into pre-commit hooks so we have some linting coverage.
I'll look into adding ruff into CI, but that'd be out of scope probably.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3409971283
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-untyped]
Review Comment:
`fsspec` is an optional dependency (the code already handles `ImportError`), but mypy will still report `import-not-found` for this import inside a `try` block. The current ignore only suppresses `import-untyped`, so type checking will still fail in environments without `fsspec`.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped]
Review Comment:
`pandas` is not a required runtime dependency of `pyarrow`, but this stub unconditionally imports it. The mypy-style `# type: ignore[...]` comment won’t suppress missing-import diagnostics in pyright, so users running pyright without pandas installed may see errors originating from pyarrow’s stubs.
##########
python/pyarrow-stubs/pyarrow/io.pyi:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Placeholder stub - complete annotations in future PR.
+# At runtime, these symbols are provided by the pyarrow.lib C extension.
+
+from typing import Any
+
+class Buffer: ...
+
+def __getattr__(name: str) -> Any: ...
Review Comment:
This stub adds a `pyarrow.io` module, but the source tree doesn’t define a corresponding runtime module (there is no `python/pyarrow/io.py` or `io.pyx`; `io.pxi` is included into `pyarrow.lib`). Shipping a stub for a non-importable module can let `import pyarrow.io` type-check while failing at runtime.
##########
python/pyarrow-stubs/pyarrow/scalar.pyi:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Placeholder stub - complete annotations in future PR.
+# At runtime, these symbols are provided by the pyarrow.lib C extension.
+
+from typing import Any
+
+class ExtensionScalar: ...
+
+def __getattr__(name: str) -> Any: ...
Review Comment:
This stub adds a `pyarrow.scalar` module, but the source tree doesn’t define a corresponding runtime module (there is no `python/pyarrow/scalar.py` or `scalar.pyx`; `scalar.pxi` is included into `pyarrow.lib`). Shipping a stub for a non-importable module can let `import pyarrow.scalar` type-check while failing at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3731910680
##########
python/pyarrow-stubs/pyarrow/error.pyi:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from types import TracebackType
+
+from typing_extensions import Self
+
+class ArrowException(Exception): ...
+class ArrowInvalid(ValueError, ArrowException): ...
+class ArrowMemoryError(MemoryError, ArrowException): ...
+class ArrowKeyError(KeyError, ArrowException): ...
+class ArrowTypeError(TypeError, ArrowException): ...
+class ArrowNotImplementedError(NotImplementedError, ArrowException): ...
+class ArrowCapacityError(ArrowException): ...
+class ArrowIndexError(IndexError, ArrowException): ...
+class ArrowSerializationError(ArrowException): ...
+
+class ArrowCancelled(ArrowException):
+ signum: int | None
+ def __init__(self, message: str, signum: int | None = None) -> None: ...
+
+ArrowIOError = IOError
+
+class StopToken: ...
+
+def enable_signal_handlers(enable: bool) -> None: ...
+
+have_signal_refcycle: bool
Review Comment:
Does this need to be typed?
(that looks like something that should have been private anyway?)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
Review Comment:
Adding the `| str` here defeats the purpose of listing the explicit possible literals?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs]
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
Review Comment:
I fully agree we want to _minimize_ overloads, but this one seems a very logical and simple one?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
Review Comment:
And also `_Tz` already includes None, so this can be:
```suggestion
unit: _Unit, tz: _Tz = None
```
?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
+def struct(
+ fields: _StructFieldsInput,
+) -> StructType: ...
+def sparse_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: Iterable[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: Sequence[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+) -> RunEndEncodedType[_RunEndType, _ValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
+ permutation: Sequence[int] | None = None,
+) -> FixedShapeTensorType[_ValueT]: ...
+def bool8() -> Bool8Type: ...
+def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
+def type_for_alias(name: str) -> DataType: ...
+def schema(
+ fields: SupportsArrowSchema | _SchemaFieldsInput,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Schema: ...
+def from_numpy_dtype(dtype: np.dtype[Any] | type | str) -> DataType: ...
+
+__all__ = [
+ "_Weakrefable",
+ "_Metadata",
+ "DataType",
+ "_BasicDataType",
+ "NullType",
+ "BoolType",
+ "UInt8Type",
+ "Int8Type",
+ "UInt16Type",
+ "Int16Type",
+ "UInt32Type",
+ "Int32Type",
+ "UInt64Type",
+ "Int64Type",
+ "Float16Type",
+ "Float32Type",
+ "Float64Type",
+ "Date32Type",
+ "Date64Type",
+ "MonthDayNanoIntervalType",
+ "StringType",
+ "LargeStringType",
+ "StringViewType",
+ "BinaryType",
+ "LargeBinaryType",
+ "BinaryViewType",
+ "TimestampType",
+ "Time32Type",
+ "Time64Type",
+ "DurationType",
+ "FixedSizeBinaryType",
+ "Decimal32Type",
+ "Decimal64Type",
+ "Decimal128Type",
+ "Decimal256Type",
+ "ListType",
+ "LargeListType",
+ "ListViewType",
+ "LargeListViewType",
+ "FixedSizeListType",
+ "DictionaryMemo",
+ "DictionaryType",
+ "MapType",
+ "StructType",
+ "UnionType",
+ "SparseUnionType",
+ "DenseUnionType",
+ "RunEndEncodedType",
+ "BaseExtensionType",
+ "ExtensionType",
+ "FixedShapeTensorType",
+ "Bool8Type",
+ "UuidType",
+ "JsonType",
+ "OpaqueType",
+ "UnknownExtensionType",
+ "register_extension_type",
+ "unregister_extension_type",
+ "KeyValueMetadata",
+ "Field",
+ "Schema",
+ "unify_schemas",
+ "field",
+ "null",
+ "bool_",
+ "uint8",
+ "int8",
+ "uint16",
+ "int16",
+ "uint32",
+ "int32",
+ "int64",
+ "uint64",
+ "timestamp",
+ "time32",
+ "time64",
+ "duration",
+ "month_day_nano_interval",
+ "date32",
+ "date64",
+ "float16",
+ "float32",
+ "float64",
+ "decimal32",
+ "decimal64",
+ "decimal128",
+ "decimal256",
+ "string",
+ "utf8",
+ "binary",
+ "large_binary",
+ "large_string",
+ "large_utf8",
+ "binary_view",
+ "string_view",
+ "list_",
+ "large_list",
+ "list_view",
+ "large_list_view",
+ "map_",
+ "dictionary",
+ "struct",
+ "sparse_union",
+ "dense_union",
+ "union",
+ "run_end_encoded",
+ "json_",
+ "uuid",
+ "fixed_shape_tensor",
+ "bool8",
+ "opaque",
+ "type_for_alias",
+ "schema",
+ "from_numpy_dtype",
+ "_Unit",
+ "_Tz",
+ "_Time32Unit",
+ "_Time64Unit",
+ "_DataTypeT",
Review Comment:
Why are those 5 exported and not others with an underscore?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
Review Comment:
Is this information of the corresponding python type ever used?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
Review Comment:
Is this one needed for typing? (I don't know if typing tools know about weakref-ability) And if we use it, shouldn't it indicate this has the `__weakref__` method?
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from _typeshed import (
+ SupportsDunderGE,
+ SupportsDunderGT,
+ SupportsDunderLE,
+ SupportsDunderLT,
+)
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+from typing_extensions import TypeAliasType
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+NumpyScalar: TypeAlias = np.generic[Any]
+
+_PyScalarT_co = TypeVar("_PyScalarT_co", bound=PyScalar, covariant=True)
+_NumpyScalarT_co = TypeVar("_NumpyScalarT_co", bound=NumpyScalar, covariant=True)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=lib.DataType, covariant=True)
+IntoArray = TypeAliasType(
+ "IntoArray",
Review Comment:
Or did you mean "InputArray" (to type keywords for array input)?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
Review Comment:
Why are the return types for those two different? (both don't change the datatype of the field)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
Review Comment:
Why `_Unit | str` and not `_Unit`? (since Unit is already a more specific set of str literals)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
Review Comment:
Does that work in practice to have the input keyword be a specific type or some generic alias, but then the output type rely on the specific input type?
What do typing linters do if they see `pa.dictionary("int64", "string")`?
(similar for map above)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
Review Comment:
Why the `default=int` here? (it is _bound_ to int already, and it does not actually have a default?)
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
Review Comment:
```suggestion
_KeyT = TypeVar("_KeyT", bound=DataType, default=DataType)
```
for better readability?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
Review Comment:
Is the `Any` needed here?
Because Field is parametrized on _DataTypeT_co, you can't do just `-> Field` ?
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from _typeshed import (
+ SupportsDunderGE,
+ SupportsDunderGT,
+ SupportsDunderLE,
+ SupportsDunderLT,
+)
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+from typing_extensions import TypeAliasType
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+NumpyScalar: TypeAlias = np.generic[Any]
+
+_PyScalarT_co = TypeVar("_PyScalarT_co", bound=PyScalar, covariant=True)
+_NumpyScalarT_co = TypeVar("_NumpyScalarT_co", bound=NumpyScalar, covariant=True)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=lib.DataType, covariant=True)
+IntoArray = TypeAliasType(
+ "IntoArray",
+ Sequence[_PyScalarT_co | None]
+ | NDArray[_NumpyScalarT_co]
+ | lib.Array[lib.Scalar[_DataTypeT_co]]
Review Comment:
I would expect:
```suggestion
| lib.Array[_DataTypeT_co]
```
Why is it parametrized on the scalar?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
Review Comment:
Could split it up with a more specific overload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4031475763
Revision: 72571d2c4e8ba0c8b1e3d292179c15f3bd15cc67
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3ea4fceeff](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3ea4fceeff)
|Task|Status|
|----|------|
|wheel-macos-monterey-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353825/job/66462415251)|
|wheel-macos-monterey-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353280/job/66462412948)|
|wheel-macos-monterey-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353367/job/66462428321)|
|wheel-macos-monterey-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905354134/job/66462416469)|
|wheel-manylinux-2-28-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905355007/job/66462419470)|
|wheel-manylinux-2-28-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905354413/job/66462416899)|
|wheel-manylinux-2-28-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905355135/job/66462420010)|
|wheel-manylinux-2-28-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353360/job/66462412976)|
|wheel-musllinux-1-2-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905354661/job/66462417755)|
|wheel-musllinux-1-2-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905354428/job/66462417282)|
|wheel-musllinux-1-2-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353262/job/66462412876)|
|wheel-musllinux-1-2-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353245/job/66462412877)|
|wheel-windows-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353821/job/66462416034)|
|wheel-windows-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22905353583/job/66462414026)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2641510816
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow.lib import BooleanArray, IntegerArray, ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | BooleanArray
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | IntegerArray
+ | ChunkedArray[Any]
+)
Review Comment:
This isn't the most exciting suggestion, but it's something that constantly furstrates me 😅
```suggestion
from pyarrow import lib
IntegerType: TypeAlias = (
lib.Int8Type
| lib.Int16Type
| lib.Int32Type
| lib.Int64Type
| lib.UInt8Type
| lib.UInt16Type
| lib.UInt32Type
| lib.UInt64Type
)
Mask: TypeAlias = (
Sequence[bool | None]
| NDArray[np.bool_]
| lib.Array[lib.Scalar[lib.BoolType]]
| ChunkedArray[Any]
)
Indices: TypeAlias = (
Sequence[int | None]
| NDArray[np.integer[Any]]
| lib.Array[lib.Scalar[IntegerType]]
| ChunkedArray[Any]
)
```
An alternative would just be `Array[Any]`.
Using the concrete subclasses requires the stubs to do a carefully choreographed dance, or the user to `typing.cast` everywhere - sadly
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow.lib import BooleanArray, IntegerArray, ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | BooleanArray
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | IntegerArray
+ | ChunkedArray[Any]
+)
Review Comment:
This isn't the most exciting suggestion, but it's something that constantly frustrates me 😅
```suggestion
from pyarrow import lib
IntegerType: TypeAlias = (
lib.Int8Type
| lib.Int16Type
| lib.Int32Type
| lib.Int64Type
| lib.UInt8Type
| lib.UInt16Type
| lib.UInt32Type
| lib.UInt64Type
)
Mask: TypeAlias = (
Sequence[bool | None]
| NDArray[np.bool_]
| lib.Array[lib.Scalar[lib.BoolType]]
| ChunkedArray[Any]
)
Indices: TypeAlias = (
Sequence[int | None]
| NDArray[np.integer[Any]]
| lib.Array[lib.Scalar[IntegerType]]
| ChunkedArray[Any]
)
```
An alternative would just be `Array[Any]`.
Using the concrete subclasses requires the stubs to do a carefully choreographed dance, or the user to `typing.cast` everywhere - sadly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3409864851
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-untyped]
Review Comment:
The `# type: ignore` here only suppresses the `import-untyped` error code. Since `fsspec` is an optional dependency (ImportError is handled), type checkers can also emit `import-not-found` when it isn't installed in the type-checking environment. Consider ignoring both codes to keep the annotation robust across environments.
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from _typeshed import (
+ SupportsDunderGE,
+ SupportsDunderGT,
+ SupportsDunderLE,
+ SupportsDunderLT,
+)
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+from typing_extensions import TypeAliasType
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
Review Comment:
In stubs, `from pyarrow import lib` will currently type `lib` as `Any` because `pyarrow/__init__.pyi` defines `__getattr__` returning `Any` and doesn't declare a `lib` attribute. Import the submodule directly so references like `lib.Int8Type` retain their (eventual) stubbed types instead of degrading to `Any`.
##########
python/pyarrow-stubs/pyarrow/io.pyi:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Placeholder stub - complete annotations in future PR.
+# At runtime, these symbols are provided by the pyarrow.lib C extension.
+
+from typing import Any
+
+class Buffer: ...
+
+def __getattr__(name: str) -> Any: ...
Review Comment:
This stub introduces a `pyarrow.io` submodule, but the Python build only compiles Cython extensions for `pyarrow.lib`, `pyarrow._compute`, `pyarrow._csv`, `pyarrow._feather`, `pyarrow._fs`, `pyarrow._json`, etc. (no `io` extension module). If `pyarrow.io` is not importable at runtime, providing a stub module for it will make type checkers accept imports that fail at runtime. Consider either adding a real `pyarrow/io.py` wrapper module (re-exporting from `pyarrow.lib`) or removing this stub and re-exporting the relevant names from the actual modules (`pyarrow/__init__.pyi` / `pyarrow/lib.pyi`).
##########
python/pyarrow-stubs/pyarrow/scalar.pyi:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Placeholder stub - complete annotations in future PR.
+# At runtime, these symbols are provided by the pyarrow.lib C extension.
+
+from typing import Any
+
+class ExtensionScalar: ...
+
+def __getattr__(name: str) -> Any: ...
Review Comment:
This stub introduces a `pyarrow.scalar` submodule, but `scalar` appears to be provided as a symbol from the `pyarrow.lib` extension (and re-exported from `pyarrow/__init__.py`), not as an importable `pyarrow.scalar` module. If `import pyarrow.scalar` is not valid at runtime, this stub module will let type checkers accept imports that will fail when executed. Consider adding a real `pyarrow/scalar.py` wrapper module (re-exporting from `pyarrow.lib`) or removing this stub module and exposing the relevant names via `pyarrow/__init__.pyi` / `pyarrow/lib.pyi`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "paddyroddy (via GitHub)" <gi...@apache.org>.
paddyroddy commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3638519840
##########
python/pyarrow-stubs/pyarrow/__init__.pyi:
##########
@@ -15,12 +15,6 @@
# specific language governing permissions and limitations
# under the License.
-"""Type stubs for PyArrow.
Review Comment:
Just noting this file was added in https://github.com/apache/arrow/pull/48618
##########
.pre-commit-config.yaml:
##########
@@ -136,11 +136,22 @@ repos:
- id: flake8
Review Comment:
Out of scope, but would suggest we move to `ruff` at some point. Faster, more frequent updates, and nicer maintainer.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
Review Comment:
Ah, so this is what I asked about elsewhere. So we're having to do different ignores per library? I've noticed that `ty` works with both `type: ignore` and `ty: ignore`.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
+def struct(
+ fields: _StructFieldsInput,
+) -> StructType: ...
+def sparse_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: Iterable[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: Sequence[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+) -> RunEndEncodedType[_RunEndType, _ValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
Review Comment:
Went to some talks at EuroPython about the new `sentinel` built-in. At some point we might want to consider, e.g. `sentinel("MISSING")`, [see this Reddit example](https://www.reddit.com/r/Python/comments/1sxmg6z/pep_661_sentinel_values_has_been_accepted_for/).
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-not-found,import-untyped]
Review Comment:
Looks like https://github.com/apache/arrow/pull/48618 introduced `mypy`, `pyright`, and `ty`. What about https://github.com/facebook/pyrefly?
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from _typeshed import (
+ SupportsDunderGE,
+ SupportsDunderGT,
+ SupportsDunderLE,
+ SupportsDunderLT,
+)
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+from typing_extensions import TypeAliasType
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+NumpyScalar: TypeAlias = np.generic[Any]
+
+_PyScalarT_co = TypeVar("_PyScalarT_co", bound=PyScalar, covariant=True)
+_NumpyScalarT_co = TypeVar("_NumpyScalarT_co", bound=NumpyScalar, covariant=True)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=lib.DataType, covariant=True)
+IntoArray = TypeAliasType(
+ "IntoArray",
Review Comment:
Is this "int to array" or "into array"?
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
+def struct(
+ fields: _StructFieldsInput,
+) -> StructType: ...
+def sparse_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: Iterable[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: Sequence[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+) -> RunEndEncodedType[_RunEndType, _ValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
+ permutation: Sequence[int] | None = None,
+) -> FixedShapeTensorType[_ValueT]: ...
+def bool8() -> Bool8Type: ...
+def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
+def type_for_alias(name: str) -> DataType: ...
+def schema(
+ fields: SupportsArrowSchema | _SchemaFieldsInput,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
Review Comment:
Only just spotted this now, but should we also have `| None`?
```suggestion
metadata: _KeyValueMetadataInput | KeyValueMetadata | None = None,
```
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-not-found,import-untyped]
Review Comment:
I haven't checked, do `type: ignore` statement codes work across these different type checkers. I've recently switched from `mypy` to `ty` and noticed some codes were different. But I have been using `ty: ignore` rather than `type: ignore` statements, which might explain things.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+@overload
+def list_(value_type: _DataTypeT | Field[_DataTypeT]) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1]
+) -> ListType[_DataTypeT]: ...
+@overload
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT],
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field[_K] | _DataTypeAliasInput,
+ item_type: _ValueT | Field[_ValueT] | _DataTypeAliasInput,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _ValueT, _Ordered]: ...
+def struct(
+ fields: _StructFieldsInput,
+) -> StructType: ...
+def sparse_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: Iterable[Field[Any]], type_codes: Sequence[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: Iterable[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: Sequence[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | _DataTypeAliasInput,
+ value_type: _ValueT | _DataTypeAliasInput,
+) -> RunEndEncodedType[_RunEndType, _ValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
+ permutation: Sequence[int] | None = None,
+) -> FixedShapeTensorType[_ValueT]: ...
+def bool8() -> Bool8Type: ...
+def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
+def type_for_alias(name: str) -> DataType: ...
+def schema(
+ fields: SupportsArrowSchema | _SchemaFieldsInput,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Schema: ...
+def from_numpy_dtype(dtype: np.dtype[Any] | type | str) -> DataType: ...
+
+__all__ = [
Review Comment:
These are usually sorted and are done automatically for me. Looks like that is coming from [RUF022](https://docs.astral.sh/ruff/rules/unsorted-dunder-all/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-5208268002
The `_types.pyi` file seems to be adding type annotations for `pyarrow/types.pxi`, is that correct?
But then why is it calles `_types.pyi` and not `types.pyi`? Is that because the types.pxi file does not actually map to a `pyarrow.types` module? (and a `pyarrow/types.pyi` file would be expected to be for that)
(but if so, why does it as a stub file for `io.pyi` and `scalar.pyi`, because that has the same layout?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "samwgoldman (via GitHub)" <gi...@apache.org>.
samwgoldman commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r4085682030
##########
python/pyarrow/fs.py:
##########
@@ -111,7 +111,7 @@ def _ensure_filesystem(filesystem, *, use_mmap=False):
else:
# handle fsspec-compatible filesystems
try:
- import fsspec
+ import fsspec # type: ignore[import-not-found,import-untyped]
Review Comment:
Hi! I'd be happy to help add Pyrefly to the CI matrix. Would it be best to wait for this change to land first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-3828710611
:warning: GitHub issue #49103 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4106228278
> @dangotbanned I'm about to have limited connectivity for 2 weeks. If you think this is ready we can ask @raulcd to do a last pass and merge, so we get it into the release.
Sorry for the delay @rok!
Just starting to dig in again now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4054994878
> Thanks for the ping @rok Yeah I will try my best to review this week.
Perfect! Looking forward :).
> 1. Are you just looking for a review of:
>
> * [`_stubs_typing.pyi`](https://github.com/apache/arrow/pull/48622/files#diff-c5785163a5f2f9960c4b1d90d2fdc8652e137433af5d88d8a3819a613c4a1b79)
> * [`_types.pyi`](https://github.com/apache/arrow/pull/48622/files#diff-9558daf1f32eda5aedc71762fbc38dc60a18634dfb05d7c77f8911e1d25537e0)
> * [`error.pyi`](https://github.com/apache/arrow/pull/48622/files#diff-83c40a21fb6ef9b823e58c402b7c67637dccfe382c49342ead56bdcb45424021)
Yes, these are the meaningful files! I've also rebased so CMake changes are no longer polluting the PR.
> 2. Have you considered using [Ruff](https://docs.astral.sh/ruff/) to help with the stubs?
I remember looking into pyi linter at some point and I forget what the outcome was. Definitely a good idea to have it here. I'll try to include it here or at least run it through the .pyi files of the PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2937037195
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
Review Comment:
I think this is hiding some ergonomics that the runtime version has:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5335-L5343
So what gets accepted by `field` would be defined by:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5761-L5852
--
## Suggestion
1. add those aliases as a `TypeAlias`
2. add another one which is `DataType | <1> | None`
3. Use that in `field(type: <2>)`
4. Then add update `struct` (and any other places) to use the same when they share the dependency 😅
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
Review Comment:
I think this is hiding some ergonomics that the runtime version has:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5335-L5343
So what gets accepted by `field` would be defined by:
https://github.com/apache/arrow/blob/a315b961cd6ab7b438d02a02f7aee3ff5c0c87c2/python/pyarrow/types.pxi#L5761-L5852
## Suggestion
1. add those aliases as a `TypeAlias`
2. add another one which is `DataType | <1> | None`
3. Use that in `field(type: <2>)`
4. Then add update `struct` (and any other places) to use the same when they share the dependency 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4031504807
@dangotbanned with packaging part mostly solved I'd like to get this and maybe https://github.com/apache/arrow/pull/49200 through review this week. Any chance you'd have time to review? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3410093447
##########
python/pyarrow-stubs/pyarrow/error.pyi:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from types import TracebackType
+
+from typing_extensions import Self
+
+class ArrowException(Exception): ...
+class ArrowInvalid(ValueError, ArrowException): ...
+class ArrowMemoryError(MemoryError, ArrowException): ...
+class ArrowKeyError(KeyError, ArrowException): ...
+class ArrowTypeError(TypeError, ArrowException): ...
+class ArrowNotImplementedError(NotImplementedError, ArrowException): ...
+class ArrowCapacityError(ArrowException): ...
+class ArrowIndexError(IndexError, ArrowException): ...
+class ArrowSerializationError(ArrowException): ...
+
+class ArrowCancelled(ArrowException):
+ signum: int | None
+ def __init__(self, message: str, signum: int | None = None) -> None: ...
+
+ArrowIOError = IOError
Review Comment:
`pyarrow.error` does not appear to be an importable runtime module (there is no `python/pyarrow/error.py` or `error.pyx`, only an `error.pxi` include that is compiled into `pyarrow.lib`). Shipping a public stub module here will make type checkers accept `import pyarrow.error`, but that import will fail at runtime.
Consider either (a) moving these exception/SignalStopHandler stubs into `pyarrow/lib.pyi` (and re-export via `pyarrow/__init__.pyi`), or (b) adding a small runtime shim module `python/pyarrow/error.py` that re-exports the symbols from `pyarrow.lib` so the module exists.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs]
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
Review Comment:
`binary()` always returns `BinaryType` when called with the default `length=-1` (see `python/pyarrow/types.pxi`), but the stub currently allows `FixedSizeBinaryType` even for the no-arg/default call. Adding overloads improves precision and prevents type checkers from accepting code that assumes a fixed-size binary type from `pa.binary()` with no arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2943841306
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
Review Comment:
@rok I'm sorry I wasn't clear on what I meant by *ergonomic*.
If you were to add an alias/multiple for the `str` versions of the type names like for example this (`Literal`, not the `type` statement bits):
- https://github.com/numpy/numpy/blob/80bcb8b03ed0b2699901e20aafaa149c9ec5a527/numpy/_typing/_char_codes.py
- https://github.com/numpy/numpy/blob/80bcb8b03ed0b2699901e20aafaa149c9ec5a527/numpy/_typing/_dtype_like.py
Then what those aliases are would be more visible, since they'll appear in autocomplete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4990558835
@paddyroddy 👋
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4031460207
@github-actions crossbow submit wheel*-cp313-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2971508518
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
Review Comment:
Not sure how many other cases you might have for this `IntoArray`.
But the building blocks to get there should be helpful anyway 🙂
```suggestion
IntegerType: TypeAlias = (
lib.Int8Type
| lib.Int16Type
| lib.Int32Type
| lib.Int64Type
| lib.UInt8Type
| lib.UInt16Type
| lib.UInt32Type
| lib.UInt64Type
)
PyScalar: TypeAlias = (
bool
| int
| float
| Decimal
| str
| bytes
| dt.date
| dt.datetime
| dt.time
| dt.timedelta
)
NumpyScalar: TypeAlias = "np.generic[Any]"
PyScalarT_co = TypeVar("PyScalarT_co", bound=PyScalar, covariant=True)
NumpyScalarT_co = TypeVar("NumpyScalarT_co", bound=NumpyScalar, covariant=True)
DataTypeT_co = TypeVar("DataTypeT_co", bound=lib.DataType, covariant=True)
IntoArray: TypeAlias = (
Sequence[PyScalarT_co | None]
| NDArray[NumpyScalarT_co]
| lib.Array[lib.Scalar[DataTypeT_co]]
| ChunkedArray[Any]
)
Mask: TypeAlias = IntoArray[bool, np.bool_, lib.BoolType]
Indices: TypeAlias = IntoArray[int, np.integer[Any], IntegerType]
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
Review Comment:
```suggestion
_BasicValueT = TypeVar(
"_BasicValueT", bound=_BasicDataType[Any], default=_BasicDataType[Any]
)
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
Review Comment:
```suggestion
def __iter__(self) -> Iterator[Field[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
Review Comment:
```suggestion
def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
Review Comment:
```suggestion
def __getitem__(self, key: str | int) -> Field[Any]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
Review Comment:
```suggestion
def flatten(self) -> list[Field[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: dict) -> Schema: ...
+ def with_metadata(self, metadata: dict) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportArrowSchema | str | Any,
+ type: _DataTypeT | str | None = None,
+ nullable: bool = True,
+ metadata: dict[Any, Any] | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | str | None = None,
+ item_type: _ValueT | Field | str | None = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | str,
+ value_type: _BasicValueT | str,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: Iterable[
+ Field[Any] | tuple[str, Field[Any] | None] | tuple[str, DataType | None]
+ ]
+ | Mapping[str, Field[Any] | DataType | None],
+) -> StructType: ...
Review Comment:
> Oh, got it. Changing, please check!
Thanks, looking much better!
I had no idea all of this was supported! 🥳
```py
>>> pa.struct({"a": pa.int64(), "b": pa.list_(pa.string(), 2)})
StructType(struct<a: int64, b: fixed_size_list<item: string>[2]>)
>>> pa.struct({"a": "int64", "b": "str", "c": "float"})
StructType(struct<a: int64, b: string, c: float>)
```
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Iterator, Sequence
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+
+Mask: TypeAlias = (
+ Sequence[bool | None]
+ | NDArray[np.bool_]
+ | lib.Array[lib.Scalar[lib.BoolType]]
+ | ChunkedArray[Any]
+)
+Indices: TypeAlias = (
+ Sequence[int | None]
+ | NDArray[np.integer[Any]]
+ | lib.Array[lib.Scalar[IntegerType]]
+ | ChunkedArray[Any]
+)
+
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportEq(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+class SupportLt(Protocol):
+ def __lt__(self, other: object, /) -> bool: ...
+
+class SupportGt(Protocol):
+ def __gt__(self, other: object, /) -> bool: ...
+
+class SupportLe(Protocol):
+ def __le__(self, other: object, /) -> bool: ...
+
+class SupportGe(Protocol):
+ def __ge__(self, other: object, /) -> bool: ...
+
+FilterTuple: TypeAlias = (
+ tuple[str, Literal["=", "==", "!="], SupportEq]
+ | tuple[str, Literal["<"], SupportLt]
+ | tuple[str, Literal[">"], SupportGt]
+ | tuple[str, Literal["<="], SupportLe]
+ | tuple[str, Literal[">="], SupportGe]
+ | tuple[str, Literal["in", "not in"], Collection]
+ | tuple[str, str, Any] # Allow general str for operator to avoid type errors
+)
+
+class Buffer(Protocol): ...
+class SupportPyBuffer(Protocol): ...
Review Comment:
> Yes, these are to be expanded in subsequent PRs.
> Would you suggest a change at this point?
No need, was just curious 😄
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
Review Comment:
```suggestion
def __iter__(self) -> Iterator[Field[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
Review Comment:
```suggestion
def _import_from_c_capsule(cls, schema: Any) -> Self: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
Review Comment:
```suggestion
def field(self, i: int | str) -> Field[Any]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
Review Comment:
```suggestion
def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
Review Comment:
```suggestion
def field(self, i: int) -> Field[Any]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
Review Comment:
Important bits here (spelling it out since this is how `cast` methods should work 😄)
- using a different `TypeVar` to the one `Field` is generic over
- the new type is invariant because it is used as a parameter and return type
```suggestion
def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
Review Comment:
```suggestion
def fields(self) -> list[Field[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
Review Comment:
If this keeps things serialized from `pyarrow.pandas_compat.construct_metadata`
```suggestion
def pandas_metadata(self) -> dict[bytes, bytes]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
Review Comment:
```suggestion
_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
Review Comment:
```suggestion
def _import_from_c_capsule(cls, schema: Any) -> Self: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
Review Comment:
Multi-part suggestion (1/3)
```suggestion
_MetadataMapping: TypeAlias = (
Mapping[bytes, bytes] | Mapping[str, str] | Mapping[bytes, str] | Mapping[str, bytes]
)
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
Review Comment:
```suggestion
def field(self, i: int | str | bytes) -> Field[Any]: ...
@deprecated("Use 'field' instead")
def field_by_name(self, name: str) -> Field[Any]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
Review Comment:
```suggestion
def __iter__(self) -> Iterator[Field[Any]]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
Review Comment:
```suggestion
def append(self, field: Field[Any]) -> Schema: ...
def insert(self, i: int, field: Field[Any]) -> Schema: ...
def remove(self, i: int) -> Schema: ...
def set(self, i: int, field: Field[Any]) -> Schema: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def with_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportsArrowSchema | str | bytes,
+ type: _DataTypeT | _DataTypeAliasInput | None = None,
+ nullable: bool = True,
+ metadata: _MetadataMapping | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | _FieldTypeInput = None,
+ item_type: _ValueT | Field | _FieldTypeInput = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
Review Comment:
```suggestion
def map_(
key_type: _K | Field[_K] | _FieldTypeInput,
item_type: _ValueT | Field[_ValueT] | _FieldTypeInput,
keys_sorted: _Ordered | None = None,
) -> MapType[_K, _ValueT, _Ordered]: ...
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def with_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportsArrowSchema | str | bytes,
+ type: _DataTypeT | _DataTypeAliasInput | None = None,
+ nullable: bool = True,
+ metadata: _MetadataMapping | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
+def map_(
+ key_type: _K | Field | _FieldTypeInput = None,
+ item_type: _ValueT | Field | _FieldTypeInput = None,
+ keys_sorted: _Ordered | None = None,
+) -> MapType[_K, _ValueT, _Ordered]: ...
+def dictionary(
+ index_type: _IndexT | _DataTypeAliasInput,
+ value_type: _BasicValueT | _DataTypeAliasInput,
+ ordered: _Ordered | None = None,
+) -> DictionaryType[_IndexT, _BasicValueT, _Ordered]: ...
+def struct(
+ fields: _StructFieldsInput,
+) -> StructType: ...
+def sparse_union(
+ child_fields: list[Field[Any]], type_codes: list[int] | None = None
+) -> SparseUnionType: ...
+def dense_union(
+ child_fields: list[Field[Any]], type_codes: list[int] | None = None
+) -> DenseUnionType: ...
+def union(
+ child_fields: list[Field[Any]],
+ mode: Literal["sparse", "dense"] | int,
+ type_codes: list[int] | None = None,
+) -> SparseUnionType | DenseUnionType: ...
+def run_end_encoded(
+ run_end_type: _RunEndType | _DataTypeAliasInput | None,
+ value_type: _BasicValueT | _DataTypeAliasInput | None,
+) -> RunEndEncodedType[_RunEndType, _BasicValueT]: ...
+def json_(storage_type: DataType = ...) -> JsonType: ...
+def uuid() -> UuidType: ...
+def fixed_shape_tensor(
+ value_type: _ValueT,
+ shape: Sequence[int],
+ dim_names: Sequence[str] | None = None,
+ permutation: Sequence[int] | None = None,
+) -> FixedShapeTensorType[_ValueT]: ...
+def bool8() -> Bool8Type: ...
+def opaque(storage_type: DataType, type_name: str, vendor_name: str) -> OpaqueType: ...
+def type_for_alias(name: Any) -> DataType: ...
+def schema(
+ fields: (
+ Iterable[Field[Any]]
+ | Iterable[tuple[str, _FieldTypeInput]]
+ | Mapping[Any, _FieldTypeInput]
+ ),
+ metadata: _SchemaMetadataInput | None = None,
Review Comment:
Multi-part suggestion (3/3)
```suggestion
metadata: _MetadataMapping | None = None,
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
Review Comment:
Multi-part suggestion (2/3)
```suggestion
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def with_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportsArrowSchema | str | bytes,
+ type: _DataTypeT | _DataTypeAliasInput | None = None,
+ nullable: bool = True,
+ metadata: _MetadataMapping | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
+def large_list(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListType[_DataTypeT]: ...
+def list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> ListViewType[_DataTypeT]: ...
+def large_list_view(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+) -> LargeListViewType[_DataTypeT]: ...
Review Comment:
You might need to do another pass on where a default `None` appears:
```suggestion
def large_list(
value_type: _DataTypeT | Field[_DataTypeT],
) -> LargeListType[_DataTypeT]: ...
def list_view(
value_type: _DataTypeT | Field[_DataTypeT],
) -> ListViewType[_DataTypeT]: ...
def large_list_view(
value_type: _DataTypeT | Field[_DataTypeT],
) -> LargeListViewType[_DataTypeT]: ...
```
Using `None` or no parameters will raise:
```py
>>> pa.list_()
TypeError: list_() takes at least 1 positional argument (0 given)
>>> pa.large_list(None)
TypeError: List requires DataType or Field
```
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str) -> Field: ...
+ def get_field_index(self, name: str) -> int: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def append(self, field: Field) -> Schema: ...
+ def insert(self, i: int, field: Field) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def with_metadata(self, metadata: _MetadataMapping) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int | None = None,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Sequence[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+def field(
+ name: SupportsArrowSchema | str | bytes,
+ type: _DataTypeT | _DataTypeAliasInput | None = None,
+ nullable: bool = True,
+ metadata: _MetadataMapping | None = None,
+) -> Field[_DataTypeT] | Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
+def large_binary() -> LargeBinaryType: ...
+def large_string() -> LargeStringType: ...
+
+large_utf8 = large_string
+
+def binary_view() -> BinaryViewType: ...
+def string_view() -> StringViewType: ...
+def list_(
+ value_type: _DataTypeT | Field[_DataTypeT] | None = None,
+ list_size: Literal[-1] | _Size | None = None,
+) -> ListType[_DataTypeT] | FixedSizeListType[_DataTypeT, _Size]: ...
Review Comment:
I'm not gonna push for overloads *everywhere*
But without this there isn't a way to create exactly `ListType` **or** `FixedSizeListType`.
```suggestion
from typing import overload
@overload
def list_(
value_type: _DataTypeT | Field[_DataTypeT], list_size: Literal[-1] | None = None
) -> ListType[_DataTypeT]: ...
@overload
def list_(
value_type: _DataTypeT | Field[_DataTypeT], list_size: _Size
) -> FixedSizeListType[_DataTypeT, _Size]: ...
```
It seems important enough to document on [`pyarrow.list_`](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html) - so probably worth being in the stubs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3381543835
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
Review Comment:
`DataType.to_pandas_dtype()` returns a NumPy dtype / scalar type and sometimes pandas extension dtypes (e.g. tz-aware). The stub currently returns `np.generic`, which is a NumPy scalar base class and doesn’t match actual return values.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
Review Comment:
`pandas` is an optional runtime dependency in pyarrow (it’s imported lazily in the implementation). Importing it unconditionally in a stub can cause type checking to fail in environments that don’t have pandas installed. Consider marking this import as ignorable for type checkers.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
Review Comment:
`Schema.pandas_metadata` can be `None` when there is no pandas metadata (implementation returns `None` if missing). The stub currently declares it as always `dict`.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,723 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType, default=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = Mapping[_StrOrBytes, _StrOrBytes]
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+_FieldTypeInput: TypeAlias = _DataTypeLike | None
+_SchemaMetadataInput: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _MetadataMapping | Any,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: DataType) -> Field: ...
+ def with_name(self, name: str) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_StructFieldTuple: TypeAlias = (
+ tuple[str, Field[Any] | None] | tuple[str, _FieldTypeInput]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _StructFieldTuple]
+ | Mapping[str, Field[Any] | _FieldTypeInput]
+)
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: str | int) -> Field: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes]: ...
Review Comment:
`Schema.metadata` returns `None` when no metadata is set (see implementation). The stub currently declares it as always `dict[bytes, bytes]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3409713771
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,727 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar(
+ "_BasicValueT", bound=_BasicDataType[Any], default=_BasicDataType[Any]
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
Review Comment:
After defining `_Size` earlier (so it can be used by `FixedSizeListType`), this second `_Size` definition becomes a redefinition and may raise a type-checking error. Remove the duplicate definition.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,727 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
Review Comment:
`TypeVar` constraints must be types; using literal `None` as a constraint (and as the default) is rejected by type checkers. Use `type(None)` (i.e., NoneType) instead so `_Tz` correctly represents `str | None`.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,727 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+_Precision_co = TypeVar("_Precision_co", default=Any, covariant=True)
+_Scale_co = TypeVar("_Scale_co", default=Any, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
Review Comment:
`FixedSizeListType` is parameterized by `_Size`, but `_Size` is defined later in the file. Type checkers (mypy/pyright) will treat `_Size` as undefined at this point and fail to analyze the stub. Define `_Size` before it is first used.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,727 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias
+
+import numpy as np
+import pandas as pd
Review Comment:
`pandas` is an optional runtime dependency for PyArrow; importing it unconditionally in a top-level stub makes type checking fail for users who have `pyarrow` installed without `pandas`. Mark the import as optional so missing `pandas` doesn't break analysis of unrelated parts of the stubs.
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+NumpyScalar: TypeAlias = "np.generic[Any]"
Review Comment:
This type alias is currently a string literal (`"np.generic[Any]"`), which type checkers won’t treat as a proper type in a `TypeAlias` assignment. That in turn makes `NumpyScalarT_co = TypeVar(..., bound=NumpyScalar)` invalid/unhelpful. Use `np.generic` (or `np.generic[Any]` if supported by the checker) directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4702780484
Finally took some time to work on this again. @dangotbanned I accepted you proposals and added some other changes. Could you please take another look and maybe we can merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-3799621647
I've rebased this on the annotation infra check PR (https://github.com/apache/arrow/pull/48618) to make sure we're on the right track.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #48622:
URL: https://github.com/apache/arrow/pull/48622#issuecomment-4064372923
Thanks for the review @dangotbanned, I did a very quick pass, will do another tomorrow and ping you again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "dangotbanned (via GitHub)" <gi...@apache.org>.
dangotbanned commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r2937043664
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,658 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Mapping, Sequence, Iterable, Iterator
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal
+
+import numpy as np
+import pandas as pd
+
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportArrowSchema, TimeUnit
+from pyarrow.io import Buffer
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ ChunkedArray,
+ ExtensionArray,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+from pyarrow.scalar import ExtensionScalar
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+
+ # Properties that exist on specific subtypes but accessed generically
+ @property
+ def list_size(self) -> int: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> np.generic: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[int], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", default=Any)
+_Scale = TypeVar("_Scale", default=Any)
+
+class Decimal32Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal64Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal128Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class Decimal256Type(FixedSizeBinaryType[Decimal], Generic[_Precision, _Scale]):
+ @property
+ def precision(self) -> _Precision: ...
+ @property
+ def scale(self) -> _Scale: ...
+
+class ListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class ListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT]: ...
+ @property
+ def value_type(self) -> _DataTypeT: ...
+ @property
+ def list_size(self) -> int: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_BasicValueT = TypeVar("_BasicValueT", bound=_BasicDataType)
+_ValueT = TypeVar("_ValueT", bound=DataType)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+
+class DictionaryType(DataType, Generic[_IndexT, _BasicValueT, _Ordered]):
+ @property
+ def ordered(self) -> _Ordered: ...
+ @property
+ def index_type(self) -> _IndexT: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_K = TypeVar("_K", bound=DataType)
+
+class MapType(DataType, Generic[_K, _ValueT, _Ordered]):
+ @property
+ def key_field(self) -> Field[_K]: ...
+ @property
+ def key_type(self) -> _K: ...
+ @property
+ def item_field(self) -> Field[_ValueT]: ...
+ @property
+ def item_type(self) -> _ValueT: ...
+ @property
+ def keys_sorted(self) -> _Ordered: ...
+
+_Size = TypeVar("_Size", default=int)
+
+class StructType(DataType):
+ def get_field_index(self, name: str) -> int: ...
+ def field(self, i: int | str) -> Field: ...
+ def get_all_field_indices(self, name: str) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field]: ...
+ def field(self, i: int) -> Field: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType, _BasicValueT]):
+ @property
+ def run_end_type(self) -> _RunEndType: ...
+ @property
+ def value_type(self) -> _BasicValueT: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array | ChunkedArray)
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT]):
+ @property
+ def value_type(self) -> _ValueT: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: ExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: Mapping[str | bytes, str | bytes]
+ | Iterable[tuple[str | bytes, str | bytes]]
+ | KeyValueMetadata
+ | None = None,
+ **kwargs: str,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: Any) -> Any: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def get_all(self, key: str) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT]):
+ def equals(self, other: Field, check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT: ...
+ def with_metadata(
+ self,
+ metadata: dict[bytes | str, bytes | str]
+ | Mapping[bytes | str, bytes | str]
+ | Any,
+ ) -> Self: ...
Review Comment:
This would be another spot for (https://github.com/apache/arrow/pull/48622#discussion_r2937040911) `TypeAlias` IIUC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r3410101207
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs]
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
+def time32(unit: _Time32Unit | str) -> Time32Type[_Time32Unit]: ...
+def time64(unit: _Time64Unit | str) -> Time64Type[_Time64Unit]: ...
+def duration(unit: _Unit | str) -> DurationType[_Unit]: ...
+def month_day_nano_interval() -> MonthDayNanoIntervalType: ...
+def date32() -> Date32Type: ...
+def date64() -> Date64Type: ...
+def float16() -> Float16Type: ...
+def float32() -> Float32Type: ...
+def float64() -> Float64Type: ...
+def decimal32(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal32Type[_Precision, _Scale | Literal[0]]: ...
+def decimal64(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal64Type[_Precision, _Scale | Literal[0]]: ...
+def decimal128(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal128Type[_Precision, _Scale | Literal[0]]: ...
+def decimal256(
+ precision: _Precision, scale: _Scale | Literal[0] = 0
+) -> Decimal256Type[_Precision, _Scale | Literal[0]]: ...
+def string() -> StringType: ...
+
+utf8 = string
+
+def binary(
+ length: Literal[-1] | int = ...,
+) -> BinaryType | FixedSizeBinaryType[bytes]: ...
Review Comment:
We're trying to minimize overloads for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #48622:
URL: https://github.com/apache/arrow/pull/48622#discussion_r4376308981
##########
python/pyarrow-stubs/pyarrow/_stubs_typing.pyi:
##########
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt
+
+from _typeshed import (
+ SupportsDunderGE,
+ SupportsDunderGT,
+ SupportsDunderLE,
+ SupportsDunderLT,
+)
+from collections.abc import Collection, Container, Iterator, Sequence, Sized
+from decimal import Decimal
+from typing import Any, Literal, Protocol, TypeAlias, TypeVar
+from typing_extensions import TypeAliasType
+
+import numpy as np
+
+from numpy.typing import NDArray
+
+from pyarrow import lib
+from pyarrow.lib import ChunkedArray
+
+ArrayLike: TypeAlias = Any
+ScalarLike: TypeAlias = Any
+Order: TypeAlias = Literal["ascending", "descending"]
+JoinType: TypeAlias = Literal[
+ "left semi",
+ "right semi",
+ "left anti",
+ "right anti",
+ "inner",
+ "left outer",
+ "right outer",
+ "full outer",
+]
+Compression: TypeAlias = Literal[
+ "gzip", "bz2", "brotli", "lz4", "lz4_frame", "lz4_raw", "zstd", "snappy"
+]
+NullEncoding: TypeAlias = Literal["mask", "encode"]
+NullSelectionBehavior: TypeAlias = Literal["drop", "emit_null"]
+TimeUnit: TypeAlias = Literal["s", "ms", "us", "ns"]
+
+IntegerType: TypeAlias = (
+ lib.Int8Type
+ | lib.Int16Type
+ | lib.Int32Type
+ | lib.Int64Type
+ | lib.UInt8Type
+ | lib.UInt16Type
+ | lib.UInt32Type
+ | lib.UInt64Type
+)
+PyScalar: TypeAlias = (
+ bool
+ | int
+ | float
+ | Decimal
+ | str
+ | bytes
+ | dt.date
+ | dt.datetime
+ | dt.time
+ | dt.timedelta
+)
+NumpyScalar: TypeAlias = np.generic[Any]
+
+_PyScalarT_co = TypeVar("_PyScalarT_co", bound=PyScalar, covariant=True)
+_NumpyScalarT_co = TypeVar("_NumpyScalarT_co", bound=NumpyScalar, covariant=True)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=lib.DataType, covariant=True)
+IntoArray = TypeAliasType(
+ "IntoArray",
+ Sequence[_PyScalarT_co | None]
+ | NDArray[_NumpyScalarT_co]
+ | lib.Array[lib.Scalar[_DataTypeT_co]]
+ | ChunkedArray[Any],
+ type_params=(_PyScalarT_co, _NumpyScalarT_co, _DataTypeT_co),
+)
+
+Mask: TypeAlias = IntoArray[bool, np.bool_, lib.BoolType]
+Indices: TypeAlias = IntoArray[int, np.integer[Any], IntegerType]
+
+_T = TypeVar("_T")
+_V = TypeVar("_V", covariant=True)
+
+SingleOrList: TypeAlias = list[_T] | _T
+
+class SupportsDunderEQ(Protocol):
+ def __eq__(self, other: object, /) -> bool: ...
+
+FilterTuple: TypeAlias = (
+ tuple[str, Literal["=", "==", "!="], SupportsDunderEQ]
+ | tuple[str, Literal["<"], SupportsDunderLT[Any]]
+ | tuple[str, Literal[">"], SupportsDunderGT[Any]]
+ | tuple[str, Literal["<="], SupportsDunderLE[Any]]
+ | tuple[str, Literal[">="], SupportsDunderGE[Any]]
+ | tuple[str, Literal["in", "not in"], Collection[Any]]
+ | tuple[str, str, Any] # Allow general str for operator to avoid type errors
+)
+
+class Buffer(Protocol): ...
+class SupportsPyBuffer(Protocol): ...
+
+class SupportsArrowStream(Protocol):
+ def __arrow_c_stream__(self, requested_schema: Any = None, /) -> Any: ...
+
+class SupportsPyArrowArray(Protocol):
+ def __arrow_array__(self, type: Any = None, /) -> Any: ...
Review Comment:
The array dispatcher calls `obj.__arrow_array__(type=type)` (`array.pxi:124`), so this parameter must accept a keyword. Making it positional-only allows implementations that fail at runtime and excludes valid keyword-only implementations. Declare `type` keyword-only in the protocol so it matches the actual calling convention.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
Review Comment:
`wrap_array()` produces an extension array rather than preserving the storage array's concrete type; chunked inputs likewise contain extension arrays in the result. Returning `_StorageT` therefore describes the wrong subclass and element type. Annotate array and chunked-array results separately instead of returning the input type variable.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
Review Comment:
Mapping is invariant in its key type, so this rejects existing `dict[str, DataType]` and `OrderedDict[str, DataType]` variables: neither is a Mapping[str | bytes, ...]. Both schema() and struct() consume this alias, so supported mapping-based construction fails type checking. Please add str-keyed and bytes-keyed mapping alternatives while retaining mixed-key support.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
Review Comment:
Inheriting `Mapping` without declaring `keys()`, `values()` and `items()` gives them view-returning types, but the runtime returns iterators (`test_types.py:1173–1175`). This rejects valid `next(metadata.items())` calls while accepting `len(metadata.keys())`, which raises `TypeError`. Explicitly model these iterator-returning methods and reconcile their signatures with the `Mapping` base.
##########
python/pyarrow-stubs/pyarrow/_types.pyi:
##########
@@ -0,0 +1,768 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime as dt # noqa: F401
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from decimal import Decimal # noqa: F401
+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd # type: ignore[import-not-found,import-untyped] # pyright: ignore[reportMissingImports,reportMissingTypeStubs] # noqa: E501
+from typing_extensions import Self, TypeVar, deprecated
+
+from pyarrow._stubs_typing import SupportsArrowSchema, TimeUnit
+from pyarrow.lib import ( # noqa: F401
+ Array,
+ Buffer,
+ ChunkedArray,
+ ExtensionArray,
+ ExtensionScalar,
+ MemoryPool,
+ MonthDayNano,
+ Table,
+)
+
+class _Weakrefable: ...
+class _Metadata(_Weakrefable): ...
+
+class DataType(_Weakrefable):
+ def field(self, i: int) -> Field[Any]: ...
+ @property
+ def id(self) -> int: ...
+ @property
+ def bit_width(self) -> int: ...
+ @property
+ def byte_width(self) -> int: ...
+ @property
+ def num_fields(self) -> int: ...
+ @property
+ def num_buffers(self) -> int: ...
+ @property
+ def has_variadic_buffers(self) -> bool: ...
+ def __hash__(self) -> int: ...
+ def equals(
+ self, other: DataType | str, *, check_metadata: bool = False
+ ) -> bool: ...
+ def to_pandas_dtype(self) -> _PandasDtype: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_AsPyType = TypeVar("_AsPyType")
+_DataTypeT = TypeVar("_DataTypeT", bound=DataType)
+_DataTypeT_co = TypeVar("_DataTypeT_co", bound=DataType, covariant=True)
+_Size = TypeVar("_Size", bound=int, default=int)
+_PandasDtype: TypeAlias = (
+ type[np.generic] | np.dtype[Any] | pd.api.extensions.ExtensionDtype
+)
+
+class _BasicDataType(DataType, Generic[_AsPyType]): ...
+class NullType(_BasicDataType[None]): ...
+class BoolType(_BasicDataType[bool]): ...
+class UInt8Type(_BasicDataType[int]): ...
+class Int8Type(_BasicDataType[int]): ...
+class UInt16Type(_BasicDataType[int]): ...
+class Int16Type(_BasicDataType[int]): ...
+class UInt32Type(_BasicDataType[int]): ...
+class Int32Type(_BasicDataType[int]): ...
+class UInt64Type(_BasicDataType[int]): ...
+class Int64Type(_BasicDataType[int]): ...
+class Float16Type(_BasicDataType[float]): ...
+class Float32Type(_BasicDataType[float]): ...
+class Float64Type(_BasicDataType[float]): ...
+class Date32Type(_BasicDataType[dt.date]): ...
+class Date64Type(_BasicDataType[dt.date]): ...
+class MonthDayNanoIntervalType(_BasicDataType[MonthDayNano]): ...
+class StringType(_BasicDataType[str]): ...
+class LargeStringType(_BasicDataType[str]): ...
+class StringViewType(_BasicDataType[str]): ...
+class BinaryType(_BasicDataType[bytes]): ...
+class LargeBinaryType(_BasicDataType[bytes]): ...
+class BinaryViewType(_BasicDataType[bytes]): ...
+
+_Unit = TypeVar("_Unit", bound=TimeUnit, default=Literal["us"])
+_Tz = TypeVar("_Tz", str, None, default=None)
+
+class TimestampType(_BasicDataType[dt.datetime], Generic[_Unit, _Tz]):
+ @property
+ def unit(self) -> _Unit: ...
+ @property
+ def tz(self) -> _Tz: ...
+
+_Time32Unit = TypeVar("_Time32Unit", bound=Literal["s", "ms"])
+
+class Time32Type(_BasicDataType[dt.time], Generic[_Time32Unit]):
+ @property
+ def unit(self) -> _Time32Unit: ...
+
+_Time64Unit = TypeVar("_Time64Unit", bound=Literal["us", "ns"])
+
+class Time64Type(_BasicDataType[dt.time], Generic[_Time64Unit]):
+ @property
+ def unit(self) -> _Time64Unit: ...
+
+class DurationType(_BasicDataType[dt.timedelta], Generic[_Unit]):
+ @property
+ def unit(self) -> _Unit: ...
+
+_FixedSizeBinaryAsPyType = TypeVar("_FixedSizeBinaryAsPyType", default=bytes)
+
+class FixedSizeBinaryType(_BasicDataType[_FixedSizeBinaryAsPyType]): ...
+
+_Precision = TypeVar("_Precision", bound=int, default=int)
+_Scale = TypeVar("_Scale", bound=int, default=int)
+_Precision_co = TypeVar("_Precision_co", bound=int, default=int, covariant=True)
+_Scale_co = TypeVar("_Scale_co", bound=int, default=int, covariant=True)
+
+class _HasPrecisionScale(Protocol[_Precision_co, _Scale_co]):
+ @property
+ def precision(self) -> _Precision_co: ...
+ @property
+ def scale(self) -> _Scale_co: ...
+
+class Decimal32Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal64Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal128Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+class Decimal256Type(
+ FixedSizeBinaryType[Decimal], _HasPrecisionScale[_Precision_co, _Scale_co]
+): ...
+
+class ListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class ListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class LargeListViewType(DataType, Generic[_DataTypeT_co]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+
+class FixedSizeListType(DataType, Generic[_DataTypeT_co, _Size]):
+ @property
+ def value_field(self) -> Field[_DataTypeT_co]: ...
+ @property
+ def value_type(self) -> _DataTypeT_co: ...
+ @property
+ def list_size(self) -> _Size: ...
+
+class DictionaryMemo(_Weakrefable): ...
+
+_IndexT = TypeVar(
+ "_IndexT",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+)
+_IndexT_co = TypeVar(
+ "_IndexT_co",
+ UInt8Type,
+ Int8Type,
+ UInt16Type,
+ Int16Type,
+ UInt32Type,
+ Int32Type,
+ UInt64Type,
+ Int64Type,
+ covariant=True,
+)
+_ValueT = TypeVar("_ValueT", bound=DataType, default=DataType)
+_ValueT_co = TypeVar("_ValueT_co", bound=DataType, default=DataType, covariant=True)
+_K = TypeVar("_K", bound=DataType, default=DataType)
+_K_co = TypeVar("_K_co", bound=DataType, default=DataType, covariant=True)
+_Ordered = TypeVar("_Ordered", Literal[True], Literal[False], default=Literal[False])
+_Ordered_co = TypeVar(
+ "_Ordered_co", Literal[True], Literal[False], default=Literal[False], covariant=True
+)
+
+class DictionaryType(DataType, Generic[_IndexT_co, _ValueT_co, _Ordered_co]):
+ @property
+ def ordered(self) -> _Ordered_co: ...
+ @property
+ def index_type(self) -> _IndexT_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+class MapType(DataType, Generic[_K_co, _ValueT_co, _Ordered_co]):
+ @property
+ def key_field(self) -> Field[_K_co]: ...
+ @property
+ def key_type(self) -> _K_co: ...
+ @property
+ def item_field(self) -> Field[_ValueT_co]: ...
+ @property
+ def item_type(self) -> _ValueT_co: ...
+ @property
+ def keys_sorted(self) -> _Ordered_co: ...
+
+class StructType(DataType):
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+
+ __getitem__ = field
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def fields(self) -> list[Field[Any]]: ...
+
+class UnionType(DataType):
+ @property
+ def mode(self) -> Literal["sparse", "dense"]: ...
+ @property
+ def type_codes(self) -> list[int]: ...
+ def __len__(self) -> int: ...
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def field(self, i: int) -> Field[Any]: ...
+
+ __getitem__ = field
+
+class SparseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["sparse"]: ...
+
+class DenseUnionType(UnionType):
+ @property
+ def mode(self) -> Literal["dense"]: ...
+
+_RunEndType = TypeVar("_RunEndType", Int16Type, Int32Type, Int64Type)
+_RunEndType_co = TypeVar(
+ "_RunEndType_co", Int16Type, Int32Type, Int64Type, covariant=True
+)
+
+class RunEndEncodedType(DataType, Generic[_RunEndType_co, _ValueT_co]):
+ @property
+ def run_end_type(self) -> _RunEndType_co: ...
+ @property
+ def value_type(self) -> _ValueT_co: ...
+
+_StorageT = TypeVar("_StorageT", bound=Array[Any] | ChunkedArray[Any])
+
+class BaseExtensionType(DataType):
+ def __arrow_ext_class__(self) -> type[ExtensionArray[Any]]: ...
+ def __arrow_ext_scalar_class__(self) -> type[ExtensionScalar]: ...
+ @property
+ def extension_name(self) -> str: ...
+ @property
+ def storage_type(self) -> DataType: ...
+ def wrap_array(self, storage: _StorageT) -> _StorageT: ...
+
+class ExtensionType(BaseExtensionType):
+ def __init__(self, storage_type: DataType, extension_name: str) -> None: ...
+ def __arrow_ext_serialize__(self) -> bytes: ...
+ @classmethod
+ def __arrow_ext_deserialize__(
+ cls, storage_type: DataType, serialized: bytes
+ ) -> Self: ...
+
+class FixedShapeTensorType(BaseExtensionType, Generic[_ValueT_co]):
+ @property
+ def value_type(self) -> _ValueT_co: ...
+ @property
+ def shape(self) -> list[int]: ...
+ @property
+ def dim_names(self) -> list[str] | None: ...
+ @property
+ def permutation(self) -> list[int] | None: ...
+
+class Bool8Type(BaseExtensionType): ...
+class UuidType(BaseExtensionType): ...
+class JsonType(BaseExtensionType): ...
+
+class OpaqueType(BaseExtensionType):
+ @property
+ def type_name(self) -> str: ...
+ @property
+ def vendor_name(self) -> str: ...
+
+class UnknownExtensionType(ExtensionType):
+ def __init__(self, storage_type: DataType, serialized: bytes) -> None: ...
+
+def register_extension_type(ext_type: BaseExtensionType) -> None: ...
+def unregister_extension_type(type_name: str) -> None: ...
+
+_FieldName: TypeAlias = str | bytes
+_StrOrBytes: TypeAlias = str | bytes
+_MetadataMapping: TypeAlias = (
+ Mapping[bytes, bytes]
+ | Mapping[str, str]
+ | Mapping[bytes, str]
+ | Mapping[str, bytes]
+)
+_MetadataIterable: TypeAlias = Iterable[tuple[_StrOrBytes, _StrOrBytes]]
+_KeyValueMetadataInput: TypeAlias = _MetadataMapping | _MetadataIterable | None
+_DataTypeAlias: TypeAlias = Literal[
+ "null",
+ "bool",
+ "boolean",
+ "i1",
+ "int8",
+ "i2",
+ "int16",
+ "i4",
+ "int32",
+ "i8",
+ "int64",
+ "u1",
+ "uint8",
+ "u2",
+ "uint16",
+ "u4",
+ "uint32",
+ "u8",
+ "uint64",
+ "f2",
+ "halffloat",
+ "float16",
+ "f4",
+ "float",
+ "float32",
+ "f8",
+ "double",
+ "float64",
+ "string",
+ "str",
+ "utf8",
+ "binary",
+ "large_string",
+ "large_str",
+ "large_utf8",
+ "large_binary",
+ "binary_view",
+ "string_view",
+ "date32",
+ "date64",
+ "date32[day]",
+ "date64[ms]",
+ "time32[s]",
+ "time32[ms]",
+ "time64[us]",
+ "time64[ns]",
+ "timestamp[s]",
+ "timestamp[ms]",
+ "timestamp[us]",
+ "timestamp[ns]",
+ "duration[s]",
+ "duration[ms]",
+ "duration[us]",
+ "duration[ns]",
+ "month_day_nano_interval",
+]
+_DataTypeAliasInput: TypeAlias = _DataTypeAlias | str
+_DataTypeLike: TypeAlias = DataType | _DataTypeAliasInput
+
+class KeyValueMetadata(_Metadata, Mapping[bytes, bytes]):
+ def __init__(
+ self,
+ __arg0__: _KeyValueMetadataInput | KeyValueMetadata = None,
+ **kwargs: str | bytes,
+ ) -> None: ...
+ def equals(self, other: KeyValueMetadata) -> bool: ...
+ def __len__(self) -> int: ...
+ def __contains__(self, /, __key: object) -> bool: ...
+ def __getitem__(self, /, __key: str | bytes) -> bytes: ...
+ def __iter__(self) -> Iterator[bytes]: ...
+ def key(self, i: int) -> bytes: ...
+ def value(self, i: int) -> bytes: ...
+ def get_all(self, key: str | bytes) -> list[bytes]: ...
+ def to_dict(self) -> dict[bytes, bytes]: ...
+
+class Field(_Weakrefable, Generic[_DataTypeT_co]):
+ def equals(self, other: Field[Any], check_metadata: bool = False) -> bool: ...
+ def __hash__(self) -> int: ...
+ @property
+ def nullable(self) -> bool: ...
+ @property
+ def name(self) -> str: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ @property
+ def type(self) -> _DataTypeT_co: ...
+ def with_metadata(
+ self,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata,
+ ) -> Self: ...
+ def remove_metadata(self) -> Self: ...
+ def with_type(self, new_type: _DataTypeT) -> Field[_DataTypeT]: ...
+ def with_name(self, name: str | bytes) -> Self: ...
+ def with_nullable(self, nullable: bool) -> Field[_DataTypeT_co]: ...
+ def flatten(self) -> list[Field[Any]]: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Self: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @classmethod
+ def _import_from_c_capsule(cls, schema: Any) -> Self: ...
+
+_FieldTuple: TypeAlias = (
+ tuple[_FieldName, _DataTypeLike]
+ | tuple[_FieldName, _DataTypeLike, bool | None]
+ | tuple[
+ _FieldName,
+ _DataTypeLike,
+ bool | None,
+ _KeyValueMetadataInput | KeyValueMetadata,
+ ]
+)
+_StructFieldsInput: TypeAlias = (
+ Iterable[Field[Any] | _FieldTuple] | Mapping[_FieldName, _DataTypeLike]
+)
+_SchemaFieldsInput: TypeAlias = _StructFieldsInput
+
+class Schema(_Weakrefable):
+ def __len__(self) -> int: ...
+ def __getitem__(self, key: int) -> Field[Any]: ...
+
+ _field = __getitem__
+ def __iter__(self) -> Iterator[Field[Any]]: ...
+ def __hash__(self) -> int: ...
+ def __sizeof__(self) -> int: ...
+ @property
+ def pandas_metadata(self) -> dict[str, Any] | None: ...
+ @property
+ def names(self) -> list[str]: ...
+ @property
+ def types(self) -> list[DataType]: ...
+ @property
+ def metadata(self) -> dict[bytes, bytes] | None: ...
+ def empty_table(self) -> Table: ...
+ def equals(self, other: Schema, check_metadata: bool = False) -> bool: ...
+ @classmethod
+ def from_pandas(
+ cls, df: pd.DataFrame, preserve_index: bool | None = None
+ ) -> Schema: ...
+ def field(self, i: int | str | bytes) -> Field[Any]: ...
+ @deprecated("Use 'field' instead")
+ def field_by_name(self, name: str | bytes) -> Field[Any] | None: ...
+ def get_field_index(self, name: str | bytes) -> int: ...
+ def get_all_field_indices(self, name: str | bytes) -> list[int]: ...
+ def append(self, field: Field[Any]) -> Schema: ...
+ def insert(self, i: int, field: Field[Any]) -> Schema: ...
+ def remove(self, i: int) -> Schema: ...
+ def set(self, i: int, field: Field[Any]) -> Schema: ...
+ @deprecated("Use 'with_metadata' instead")
+ def add_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def with_metadata(
+ self, metadata: _KeyValueMetadataInput | KeyValueMetadata
+ ) -> Schema: ...
+ def serialize(self, memory_pool: MemoryPool | None = None) -> Buffer: ...
+ def remove_metadata(self) -> Schema: ...
+ def to_string(
+ self,
+ truncate_metadata: bool = True,
+ show_field_metadata: bool = True,
+ show_schema_metadata: bool = True,
+ element_size_limit: int = 100,
+ ) -> str: ...
+ def _export_to_c(self, out_ptr: int) -> None: ...
+ @classmethod
+ def _import_from_c(cls, in_ptr: int) -> Schema: ...
+ def __arrow_c_schema__(self) -> Any: ...
+ @staticmethod
+ def _import_from_c_capsule(schema: Any) -> Schema: ...
+
+def unify_schemas(
+ schemas: Iterable[Schema],
+ *,
+ promote_options: Literal["default", "permissive"] = "default",
+) -> Schema: ...
+@overload
+def field(
+ name: SupportsArrowSchema,
+ type: None = None,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeT,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[_DataTypeT]: ...
+@overload
+def field(
+ name: _FieldName,
+ type: _DataTypeAliasInput,
+ nullable: bool | None = None,
+ metadata: _KeyValueMetadataInput | KeyValueMetadata = None,
+) -> Field[Any]: ...
+def null() -> NullType: ...
+def bool_() -> BoolType: ...
+def uint8() -> UInt8Type: ...
+def int8() -> Int8Type: ...
+def uint16() -> UInt16Type: ...
+def int16() -> Int16Type: ...
+def uint32() -> UInt32Type: ...
+def int32() -> Int32Type: ...
+def int64() -> Int64Type: ...
+def uint64() -> UInt64Type: ...
+def timestamp(
+ unit: _Unit | str, tz: _Tz | None = None
+) -> TimestampType[_Unit, _Tz]: ...
Review Comment:
The timezone is constrained to str or None here, but timestamp() in types.pxi also accepts datetime.tzinfo objects and bytes, converting them to timezone strings. Valid calls such as `timestamp('us', tz=datetime.timezone.utc)` are therefore rejected by this stub. Please include those inputs and describe their resulting `.tz` as str.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org