You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/04 23:15:56 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
xinrong-databricks opened a new pull request #34174:
URL: https://github.com/apache/spark/pull/34174
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/sql/types.py
### Why are the changes needed?
Current stub files cannot support type checking for the function body. Inline type hints can type check the function body.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942908313
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34174:
URL: https://github.com/apache/spark/pull/34174#discussion_r727570315
##########
File path: python/pyspark/sql/types.py
##########
@@ -95,12 +110,13 @@ def fromInternal(self, obj):
class DataTypeSingleton(type):
"""Metaclass for DataType"""
- _instances = {}
+ _instances: Dict = {}
Review comment:
How about:
```py
_instances: Dict[Type["DataTypeSingleton"], "DataTypeSingleton"] = {}
```
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6429,14 +6429,14 @@ def select_dtypes(
include_spark_type = []
for inc in include_list:
try:
- include_spark_type.append(_parse_datatype_string(inc))
+ include_spark_type.append(_parse_datatype_string(cast(str, inc)))
Review comment:
Shall we rather specify the type of `include_list`?
at line 6408-6411:
```py
include_list: List[str]
if not is_list_like(include):
include_list = [cast(str, include)] if include is not None else []
else:
include_list = list(include)
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1446,25 +1517,25 @@ def verify_struct(obj):
verify_value = verify_struct
else:
- def verify_default(obj):
+ def verify_default(obj: Any) -> None:
assert_acceptable_types(obj)
verify_acceptable_types(obj)
verify_value = verify_default
- def verify(obj):
+ def verify(obj: Any) -> None:
if not verify_nullability(obj):
verify_value(obj)
return verify
# This is used to unpickle a Row from JVM
-def _create_row_inbound_converter(dataType):
+def _create_row_inbound_converter(dataType: DataType) -> Callable:
return lambda *a: dataType.fromInternal(a)
-def _create_row(fields, values):
+def _create_row(fields: Any, values: Any) -> "Row":
Review comment:
I guess:
```py
def _create_row(fields: List[str], values: List[Any]) -> "Row":
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1224,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(a.elementType, b.elementType, # type: ignore[attr-defined]
name='element in array %s' % name), True)
elif isinstance(a, MapType):
- return MapType(_merge_type(a.keyType, b.keyType, name='key of map %s' % name),
- _merge_type(a.valueType, b.valueType, name='value of map %s' % name),
- True)
+ return MapType(
+ _merge_type(a.keyType, b.keyType, # type: ignore[attr-defined]
+ name='key of map %s' % name),
+ _merge_type(a.valueType, b.valueType, # type: ignore[attr-defined]
+ name='value of map %s' % name),
Review comment:
I guess we can also avoid `ignore[attr-defined]` her.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1015,7 +1065,7 @@ def _int_size_to_type(size):
size = ctypes.sizeof(_array_signed_int_typecode_ctype_mappings[_typecode]) * 8
dt = _int_size_to_type(size)
if dt is not None:
- _array_type_mappings[_typecode] = dt
+ _array_type_mappings[_typecode] = dt # type: ignore[assignment]
Review comment:
Shall we rather specify the type of `_array_type_mappings`?
```py
_array_type_mappings: Dict[str, Type[DataType]] = {
...
}
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1150,7 +1213,7 @@ def _merge_type(a, b, name=None):
# same type
if isinstance(a, StructType):
- nfs = dict((f.name, f.dataType) for f in b.fields)
+ nfs = dict((f.name, f.dataType) for f in b.fields) # type: ignore[attr-defined]
Review comment:
We should use `cast(StructType, b).fields`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -522,7 +546,27 @@ def __init__(self, fields=None):
self._needConversion = [f.needConversion() for f in self]
self._needSerializeAnyField = any(self._needConversion)
- def add(self, field, data_type=None, nullable=True, metadata=None):
+ @overload
+ def add(
+ self,
+ field: str,
+ data_type: Union[str, DataType],
+ nullable: bool = True,
+ metadata: Optional[Dict[str, Any]] = None,
+ ) -> "StructType":
+ ...
+
+ @overload
+ def add(self, field: StructField) -> "StructType":
+ ...
+
+ def add(
+ self,
+ field: Union[str, StructField],
+ data_type: Optional[Union[str, DataType]] = None,
+ nullable: Optional[bool] = True,
Review comment:
`nullable: bool = True`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -439,7 +457,13 @@ class StructField(DataType):
False
"""
- def __init__(self, name, dataType, nullable=True, metadata=None):
+ def __init__(
+ self,
+ name: str,
+ dataType: DataType,
+ nullable: Optional[bool] = True,
Review comment:
`nullable: bool = True`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -232,18 +248,18 @@ class DecimalType(FractionalType):
the number of digits on right side of dot. (default: 0)
"""
- def __init__(self, precision=10, scale=0):
+ def __init__(self, precision: int = 10, scale: int = 0) -> None:
Review comment:
nit: I don't think we need `-> None` for the initializer.
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6429,14 +6429,14 @@ def select_dtypes(
include_spark_type = []
for inc in include_list:
try:
- include_spark_type.append(_parse_datatype_string(inc))
+ include_spark_type.append(_parse_datatype_string(cast(str, inc)))
except:
pass
exclude_spark_type = []
for exc in exclude_list:
try:
- exclude_spark_type.append(_parse_datatype_string(exc))
+ exclude_spark_type.append(_parse_datatype_string(cast(str, exc)))
Review comment:
ditto.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1137,31 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[str] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
if isinstance(row, dict):
items = sorted(row.items())
elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
- items = zip(row.__fields__, tuple(row))
+ items: zip[Tuple[Any, Any]] = zip( # type: ignore[no-redef]
Review comment:
Could you explicitly specify a proper type of `items` beforehand and avoid using `ignore[no-redef]`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -677,7 +721,7 @@ def fromInternal(self, obj):
values = [f.fromInternal(v) if c else v
for f, v, c in zip(self.fields, obj, self._needConversion)]
else:
- values = obj
+ values: Tuple = obj # type: ignore[no-redef]
Review comment:
```py
values: Tuple
if self._needSerializeAnyField:
...
else:
...
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1055,7 +1109,7 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
struct = StructType()
for key, value in obj.items():
if key is not None and value is not None:
- struct.add(
+ struct.add( # type: ignore[call-overload]
Review comment:
Do we need this?
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1224,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(a.elementType, b.elementType, # type: ignore[attr-defined]
Review comment:
Could you avoid using `ignore[attr-defined]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34174:
URL: https://github.com/apache/spark/pull/34174#discussion_r728513483
##########
File path: python/pyspark/sql/types.py
##########
@@ -786,22 +833,22 @@ def fromJson(cls, json):
UDT = getattr(m, pyClass)
return UDT()
- def __eq__(self, other):
+ def __eq__(self, other: Any) -> bool:
return type(self) == type(other)
_atomic_types = [StringType, BinaryType, BooleanType, DecimalType, FloatType, DoubleType,
ByteType, ShortType, IntegerType, LongType, DateType, TimestampType,
TimestampNTZType, NullType]
-_all_atomic_types = dict((t.typeName(), t) for t in _atomic_types)
-_all_complex_types = dict((v.typeName(), v)
+_all_atomic_types = dict((t.typeName(), t) for t in _atomic_types) # type: ignore[attr-defined]
+_all_complex_types = dict((v.typeName(), v) # type: ignore[attr-defined]
for v in [ArrayType, MapType, StructType])
Review comment:
How about:
```py
_atomic_types: List[Type[DataType]] = ...
_all_atomic_types: Dict[str, Type[DataType]] = dict((t.typeName(), t) for t in _atomic_types)
_complex_types: List[Type[DataType]] = [ArrayType, MapType, StructType]
_all_complex_types: Dict[str, Type[DataType]] = dict((v.typeName(), v) for v in _complex_types)
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1029,10 +1082,14 @@ def _int_size_to_type(size):
# Type code 'u' in Python's array is deprecated since version 3.3, and will be
# removed in version 4.0. See: https://docs.python.org/3/library/array.html
if sys.version_info[0] < 4:
- _array_type_mappings['u'] = StringType
+ _array_type_mappings['u'] = StringType # type: ignore[assignment]
Review comment:
Do we still need this?
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1140,30 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[List[str]] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
+ items: Union[zip[Tuple[Any, Any]], List[Tuple[Any, Any]]]
if isinstance(row, dict):
items = sorted(row.items())
elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
- items = zip(row.__fields__, tuple(row))
+ items = zip(row.__fields__, tuple(row)) # type: ignore[union-attr]
elif hasattr(row, "_fields"): # namedtuple
- items = zip(row._fields, tuple(row))
+ items = zip(row._fields, tuple(row)) # type: ignore[union-attr]
else:
if names is None:
- names = ['_%d' % i for i in range(1, len(row) + 1)]
+ names = [
+ '_%d' % i for i in range(1, len(row) + 1)] # type: ignore[no-redef, assignment]
elif len(names) < len(row):
- names.extend('_%d' % i for i in range(len(names) + 1, len(row) + 1))
- items = zip(names, row)
+ names.extend( # type: ignore[attr-defined]
+ '_%d' % i for i in range(len(names) + 1, len(row) + 1))
+ items = zip(names, row) # type: ignore[arg-type, assignment]
Review comment:
ditto.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1140,30 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[List[str]] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
+ items: Union[zip[Tuple[Any, Any]], List[Tuple[Any, Any]]]
if isinstance(row, dict):
items = sorted(row.items())
elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
- items = zip(row.__fields__, tuple(row))
+ items = zip(row.__fields__, tuple(row)) # type: ignore[union-attr]
elif hasattr(row, "_fields"): # namedtuple
- items = zip(row._fields, tuple(row))
+ items = zip(row._fields, tuple(row)) # type: ignore[union-attr]
else:
if names is None:
- names = ['_%d' % i for i in range(1, len(row) + 1)]
+ names = [
+ '_%d' % i for i in range(1, len(row) + 1)] # type: ignore[no-redef, assignment]
elif len(names) < len(row):
- names.extend('_%d' % i for i in range(len(names) + 1, len(row) + 1))
- items = zip(names, row)
+ names.extend( # type: ignore[attr-defined]
Review comment:
ditto.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1140,30 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[List[str]] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
+ items: Union[zip[Tuple[Any, Any]], List[Tuple[Any, Any]]]
if isinstance(row, dict):
items = sorted(row.items())
elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
- items = zip(row.__fields__, tuple(row))
+ items = zip(row.__fields__, tuple(row)) # type: ignore[union-attr]
elif hasattr(row, "_fields"): # namedtuple
- items = zip(row._fields, tuple(row))
+ items = zip(row._fields, tuple(row)) # type: ignore[union-attr]
else:
if names is None:
- names = ['_%d' % i for i in range(1, len(row) + 1)]
+ names = [
+ '_%d' % i for i in range(1, len(row) + 1)] # type: ignore[no-redef, assignment]
Review comment:
Do we need the `ignore`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1140,30 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[List[str]] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
+ items: Union[zip[Tuple[Any, Any]], List[Tuple[Any, Any]]]
Review comment:
How about `items: Iterable[Tuple[str, Any]]`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -1128,7 +1193,7 @@ def _has_nulltype(dt):
return isinstance(dt, NullType)
-def _merge_type(a, b, name=None):
+def _merge_type(a: DataType, b: DataType, name: Optional[str] = None) -> DataType:
Review comment:
Shall we define overloads?
```py
@overload
def _merge_type(a: StructType, b: StructType, name: Optional[str] = ...) -> StructType:
...
@overload
def _merge_type(a: ArrayType, b: ArrayType, name: Optional[str] = ...) -> ArrayType:
...
@overload
def _merge_type(a: MapType, b: MapType, name: Optional[str] = ...) -> MapType:
...
@overload
def _merge_type(a: DataType, b: DataType, name: Optional[str] = ...) -> DataType:
...
```
Then we can remove the changes in `session.py`.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1226,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(cast(ArrayType, a).elementType, cast(ArrayType, b).elementType,
Review comment:
nit: we don't need `cast` for `a` here.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1226,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(cast(ArrayType, a).elementType, cast(ArrayType, b).elementType,
name='element in array %s' % name), True)
elif isinstance(a, MapType):
- return MapType(_merge_type(a.keyType, b.keyType, name='key of map %s' % name),
- _merge_type(a.valueType, b.valueType, name='value of map %s' % name),
- True)
+ return MapType(
+ _merge_type(cast(MapType, a).keyType, cast(MapType, b).keyType,
+ name='key of map %s' % name),
+ _merge_type(cast(MapType, a).valueType, cast(MapType, b).valueType,
+ name='value of map %s' % name),
Review comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-944505730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942908313
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936785222
**[Test build #143887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143887/testReport)** for PR 34174 at commit [`b06a932`](https://github.com/apache/spark/commit/b06a932397382fcf0011b413f2fd5b04adf014d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936887616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143887/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942795995
**[Test build #144216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144216/testReport)** for PR 34174 at commit [`fd764a0`](https://github.com/apache/spark/commit/fd764a0ebb9eb3d3412ab39ed1fdcab0f3e48971).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933943409
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143822/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933984734
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936998044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936998044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933957747
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-944567986
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-938292901
Looks fine for my quick glance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942931068
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48706/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933988961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937102212
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942895068
**[Test build #144225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144225/testReport)** for PR 34174 at commit [`fb8fe31`](https://github.com/apache/spark/commit/fb8fe311a57e4139b73f2a095cb2cd3243e6636f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942838957
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942851772
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933943226
**[Test build #143822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143822/testReport)** for PR 34174 at commit [`033c2c8`](https://github.com/apache/spark/commit/033c2c8378700675750261b31f5c1f8f7f3c4e8d).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936785222
**[Test build #143887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143887/testReport)** for PR 34174 at commit [`b06a932`](https://github.com/apache/spark/commit/b06a932397382fcf0011b413f2fd5b04adf014d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-944505730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936891212
**[Test build #143892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143892/testReport)** for PR 34174 at commit [`4bea99a`](https://github.com/apache/spark/commit/4bea99a52ec2c9a10f225a46c0c3b77f5bb29772).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34174:
URL: https://github.com/apache/spark/pull/34174#discussion_r730001470
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -308,9 +308,11 @@ def schema(self) -> StructType:
>>> df.schema
StructType(List(StructField(age,IntegerType,true),StructField(name,StringType,true)))
"""
+ self._schema: StructType # type: ignore[no-redef]
Review comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942919811
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48706/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-944474289
**[Test build #144310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144310/testReport)** for PR 34174 at commit [`df0b717`](https://github.com/apache/spark/commit/df0b71763e89e65e1542eb077232f2b6522d481c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942789879
**[Test build #144216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144216/testReport)** for PR 34174 at commit [`fd764a0`](https://github.com/apache/spark/commit/fd764a0ebb9eb3d3412ab39ed1fdcab0f3e48971).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936948459
**[Test build #143892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143892/testReport)** for PR 34174 at commit [`4bea99a`](https://github.com/apache/spark/commit/4bea99a52ec2c9a10f225a46c0c3b77f5bb29772).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936891212
**[Test build #143892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143892/testReport)** for PR 34174 at commit [`4bea99a`](https://github.com/apache/spark/commit/4bea99a52ec2c9a10f225a46c0c3b77f5bb29772).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936960985
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933988961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933937667
**[Test build #143822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143822/testReport)** for PR 34174 at commit [`033c2c8`](https://github.com/apache/spark/commit/033c2c8378700675750261b31f5c1f8f7f3c4e8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933937667
**[Test build #143822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143822/testReport)** for PR 34174 at commit [`033c2c8`](https://github.com/apache/spark/commit/033c2c8378700675750261b31f5c1f8f7f3c4e8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937385018
cc @zero323 @itholic @ueshin FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942884621
**[Test build #144225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144225/testReport)** for PR 34174 at commit [`fb8fe31`](https://github.com/apache/spark/commit/fb8fe311a57e4139b73f2a095cb2cd3243e6636f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937077350
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942851772
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936887616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143887/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937385018
cc @zero323 @itholic @ueshin FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937102212
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-933943409
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143822/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942789879
**[Test build #144216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144216/testReport)** for PR 34174 at commit [`fd764a0`](https://github.com/apache/spark/commit/fd764a0ebb9eb3d3412ab39ed1fdcab0f3e48971).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942796133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936842439
**[Test build #143887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143887/testReport)** for PR 34174 at commit [`b06a932`](https://github.com/apache/spark/commit/b06a932397382fcf0011b413f2fd5b04adf014d0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937077350
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942884621
**[Test build #144225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144225/testReport)** for PR 34174 at commit [`fb8fe31`](https://github.com/apache/spark/commit/fb8fe311a57e4139b73f2a095cb2cd3243e6636f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #34174:
URL: https://github.com/apache/spark/pull/34174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936979919
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942902472
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48706/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942808586
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-936877534
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937102212
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942931068
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48706/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-937102212
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34174:
URL: https://github.com/apache/spark/pull/34174#discussion_r727570315
##########
File path: python/pyspark/sql/types.py
##########
@@ -95,12 +110,13 @@ def fromInternal(self, obj):
class DataTypeSingleton(type):
"""Metaclass for DataType"""
- _instances = {}
+ _instances: Dict = {}
Review comment:
How about:
```py
_instances: Dict[Type["DataTypeSingleton"], "DataTypeSingleton"] = {}
```
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6429,14 +6429,14 @@ def select_dtypes(
include_spark_type = []
for inc in include_list:
try:
- include_spark_type.append(_parse_datatype_string(inc))
+ include_spark_type.append(_parse_datatype_string(cast(str, inc)))
Review comment:
Shall we rather specify the type of `include_list`?
at line 6408-6411:
```py
include_list: List[str]
if not is_list_like(include):
include_list = [cast(str, include)] if include is not None else []
else:
include_list = list(include)
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1446,25 +1517,25 @@ def verify_struct(obj):
verify_value = verify_struct
else:
- def verify_default(obj):
+ def verify_default(obj: Any) -> None:
assert_acceptable_types(obj)
verify_acceptable_types(obj)
verify_value = verify_default
- def verify(obj):
+ def verify(obj: Any) -> None:
if not verify_nullability(obj):
verify_value(obj)
return verify
# This is used to unpickle a Row from JVM
-def _create_row_inbound_converter(dataType):
+def _create_row_inbound_converter(dataType: DataType) -> Callable:
return lambda *a: dataType.fromInternal(a)
-def _create_row(fields, values):
+def _create_row(fields: Any, values: Any) -> "Row":
Review comment:
I guess:
```py
def _create_row(fields: List[str], values: List[Any]) -> "Row":
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1224,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(a.elementType, b.elementType, # type: ignore[attr-defined]
name='element in array %s' % name), True)
elif isinstance(a, MapType):
- return MapType(_merge_type(a.keyType, b.keyType, name='key of map %s' % name),
- _merge_type(a.valueType, b.valueType, name='value of map %s' % name),
- True)
+ return MapType(
+ _merge_type(a.keyType, b.keyType, # type: ignore[attr-defined]
+ name='key of map %s' % name),
+ _merge_type(a.valueType, b.valueType, # type: ignore[attr-defined]
+ name='value of map %s' % name),
Review comment:
I guess we can also avoid `ignore[attr-defined]` her.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1015,7 +1065,7 @@ def _int_size_to_type(size):
size = ctypes.sizeof(_array_signed_int_typecode_ctype_mappings[_typecode]) * 8
dt = _int_size_to_type(size)
if dt is not None:
- _array_type_mappings[_typecode] = dt
+ _array_type_mappings[_typecode] = dt # type: ignore[assignment]
Review comment:
Shall we rather specify the type of `_array_type_mappings`?
```py
_array_type_mappings: Dict[str, Type[DataType]] = {
...
}
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1150,7 +1213,7 @@ def _merge_type(a, b, name=None):
# same type
if isinstance(a, StructType):
- nfs = dict((f.name, f.dataType) for f in b.fields)
+ nfs = dict((f.name, f.dataType) for f in b.fields) # type: ignore[attr-defined]
Review comment:
We should use `cast(StructType, b).fields`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -522,7 +546,27 @@ def __init__(self, fields=None):
self._needConversion = [f.needConversion() for f in self]
self._needSerializeAnyField = any(self._needConversion)
- def add(self, field, data_type=None, nullable=True, metadata=None):
+ @overload
+ def add(
+ self,
+ field: str,
+ data_type: Union[str, DataType],
+ nullable: bool = True,
+ metadata: Optional[Dict[str, Any]] = None,
+ ) -> "StructType":
+ ...
+
+ @overload
+ def add(self, field: StructField) -> "StructType":
+ ...
+
+ def add(
+ self,
+ field: Union[str, StructField],
+ data_type: Optional[Union[str, DataType]] = None,
+ nullable: Optional[bool] = True,
Review comment:
`nullable: bool = True`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -439,7 +457,13 @@ class StructField(DataType):
False
"""
- def __init__(self, name, dataType, nullable=True, metadata=None):
+ def __init__(
+ self,
+ name: str,
+ dataType: DataType,
+ nullable: Optional[bool] = True,
Review comment:
`nullable: bool = True`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -232,18 +248,18 @@ class DecimalType(FractionalType):
the number of digits on right side of dot. (default: 0)
"""
- def __init__(self, precision=10, scale=0):
+ def __init__(self, precision: int = 10, scale: int = 0) -> None:
Review comment:
nit: I don't think we need `-> None` for the initializer.
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6429,14 +6429,14 @@ def select_dtypes(
include_spark_type = []
for inc in include_list:
try:
- include_spark_type.append(_parse_datatype_string(inc))
+ include_spark_type.append(_parse_datatype_string(cast(str, inc)))
except:
pass
exclude_spark_type = []
for exc in exclude_list:
try:
- exclude_spark_type.append(_parse_datatype_string(exc))
+ exclude_spark_type.append(_parse_datatype_string(cast(str, exc)))
Review comment:
ditto.
##########
File path: python/pyspark/sql/types.py
##########
@@ -1083,22 +1137,31 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
raise TypeError("not supported type: %s" % type(obj))
-def _infer_schema(row, names=None, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
+def _infer_schema(
+ row: Any,
+ names: Optional[str] = None,
+ infer_dict_as_struct: bool = False,
+ prefer_timestamp_ntz: bool = False,
+) -> StructType:
"""Infer the schema from dict/namedtuple/object"""
if isinstance(row, dict):
items = sorted(row.items())
elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
- items = zip(row.__fields__, tuple(row))
+ items: zip[Tuple[Any, Any]] = zip( # type: ignore[no-redef]
Review comment:
Could you explicitly specify a proper type of `items` beforehand and avoid using `ignore[no-redef]`?
##########
File path: python/pyspark/sql/types.py
##########
@@ -677,7 +721,7 @@ def fromInternal(self, obj):
values = [f.fromInternal(v) if c else v
for f, v, c in zip(self.fields, obj, self._needConversion)]
else:
- values = obj
+ values: Tuple = obj # type: ignore[no-redef]
Review comment:
```py
values: Tuple
if self._needSerializeAnyField:
...
else:
...
```
##########
File path: python/pyspark/sql/types.py
##########
@@ -1055,7 +1109,7 @@ def _infer_type(obj, infer_dict_as_struct=False, prefer_timestamp_ntz=False):
struct = StructType()
for key, value in obj.items():
if key is not None and value is not None:
- struct.add(
+ struct.add( # type: ignore[call-overload]
Review comment:
Do we need this?
##########
File path: python/pyspark/sql/types.py
##########
@@ -1161,18 +1224,21 @@ def _merge_type(a, b, name=None):
return StructType(fields)
elif isinstance(a, ArrayType):
- return ArrayType(_merge_type(a.elementType, b.elementType,
+ return ArrayType(_merge_type(a.elementType, b.elementType, # type: ignore[attr-defined]
Review comment:
Could you avoid using `ignore[attr-defined]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-942796133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34174: [SPARK-36910][PYTHON] Inline type hints for python/pyspark/sql/types.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34174:
URL: https://github.com/apache/spark/pull/34174#issuecomment-944474289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org