You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/14 01:05:16 UTC
[GitHub] [spark] ueshin commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
ueshin commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707826582
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -84,11 +87,24 @@ def __repr__(self) -> str:
class DataFrameType(object):
def __init__(
- self, dtypes: List[Dtype], spark_types: List[types.DataType], names: List[Optional[str]]
+ self,
+ index_name: Optional[str],
+ index_dtype: Dtype,
+ data_dtypes: List[Dtype],
+ data_names: List[Optional[str]],
+ spark_types: List[types.DataType],
):
from pyspark.pandas.internal import InternalField
from pyspark.pandas.utils import name_like_string
+ names = [index_name] + data_names if index_dtype is not None else data_names
+ dtypes = [index_dtype] + data_dtypes if index_dtype is not None else data_dtypes
+
+ self.index_name = index_name
+ self.index_dtype = index_dtype
+ self.data_dtypes = data_dtypes
+ self.data_names = data_names
Review comment:
I guess we should manage `index_field` and `data_fields` instead?
We can do it later, though.
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
+
+ Typing data columns only:
+
+ >>> pd.DataFrame[float, float]
Review comment:
I'm wondering why we are using `pd` instead of `ps`?
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
Review comment:
What's this example for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org