You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/10 07:46:02 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
HyukjinKwon opened a new pull request #33954:
URL: https://github.com/apache/spark/pull/33954
### What changes were proposed in this pull request?
This PR proposes new syntax to specify the index type and name in pandas API on Spark. This is a base work for SPARK-36707.
More specifically, users now can use the type hints when typing as below:
```
pd.DataFrame[int, [int, int]]
pd.DataFrame[pdf.index.dtype, pdf.dtypes]
pd.DataFrame[("index", int), [("id", int), ("A", int)]]
pd.DataFrame[(pdf.index.name, pdf.index.dtype), zip(pdf.columns, pdf.dtypes)]
```
Note that the types of `[("id", int), ("A", int)]` or `("index", int)` are matched to how you provide a compound NumPy type (see also https://numpy.org/doc/stable/user/basics.rec.html#introduction).
Therefore, the syntax will be:
**Without index:**
```
pd.DataFrame[type, type, ...]
pd.DataFrame[name: type, name: type, ...]
pd.DataFrame[dtypes instance]
pd.DataFrame[zip(names, types)]
```
**With index:**
```
pd.DataFrame[index_type, [type, ...]]
pd.DataFrame[(index_name, index_type), [(name, type), ...]]
pd.DataFrame[dtype instance, dtypes instance]
pd.DataFrame[(index_name, index_type), zip(names, types)]
```
### Why are the changes needed?
Currently, there is no way to specify the type hint for index type - the type hints are converted to return type of pandas UDFs internally. Therefore, we always attach default index which degrade performance:
```python
>>> def transform(pdf) -> pd.DataFrame[int, int]:
... pdf['A'] = pdf.id + 1
... return pdf
...
>>> ks.range(5).koalas.apply_batch(transform)
```
```
c0 c1
0 0 1
1 1 2
2 2 3
3 3 4
4 4 5
```
The [default index](https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type) (for the first column that looks unnamed) is attached when the type hint is specified. For better performance, we should have a way to work around, see also [Specify the index column in conversion from Spark DataFrame to Koalas DataFrame](https://koalas.readthedocs.io/en/latest/user_guide/best_practices.html#specify-the-index-column-in-conversion-from-spark-dataframe-to-koalas-dataframe).
Note that this still remains as experimental because Python itself yet doesn't support such kind of typing out of the box. Once pandas completes typing support like NumPy did in `numpy.typing`, we should implement Koalas typing package, and migrate to it with leveraging pandas' typing way.
### Does this PR introduce _any_ user-facing change?
No, this PR does not yet cause user-facing behavior.
### How was this patch tested?
Unittests were added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916734929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916731726
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918796357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917742920
When the index type is not specified, the default index would be attached by default for the output DataFrame. Using default index usually brings some performance penalty. See the pseudo code below.
Assume we're using distributed sequence index (https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type), and we perform an panda API such as:
```python
df.apply(func)
```
Internally the Spark operations as below would be performed (pseudo code)
**Without index type hint:**
```python
spark_df = df.to_spark()
# compute date part
spark_df.select(udf(func, "data column types without index"))
# compute index part
spark.createDataFrame(spark_df.rdd.zipWithIndex().map(lambda p: p[1], p[0])) # attach index
```
**With index type hint:**
```python
spark_df = df.to_spark()
spark_df.select(udf(func), "column types with both data and index")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918774029
**[Test build #143233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143233/testReport)** for PR 33954 at commit [`23c73c4`](https://github.com/apache/spark/commit/23c73c446f6ed569e6d90e4924291a0163a99713).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917742920
When the index type is not specified, the default index would be attached by default for the output DataFrame. Using default index usually brings some performance penalty. See the pseudo code below.
Assume we're using distributed sequence index (https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type), and we perform an panda API such as:
```python
df.apply(func)
```
Internally the Spark operations as below are performed (pseudo code)
**Without index type hint:**
```python
spark_df = df.to_spark()
# compute date part
spark_df = spark_df.select(udf(func, "data column types without index"))
# compute index part
spark.createDataFrame(spark_df.rdd.zipWithIndex().map(lambda p: p[1], p[0])) # attach index
```
**With index type hint:**
```python
spark_df = df.to_spark()
spark_df.select(udf(func, "column types with both data and index"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918796356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916734929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707826582
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -84,11 +87,24 @@ def __repr__(self) -> str:
class DataFrameType(object):
def __init__(
- self, dtypes: List[Dtype], spark_types: List[types.DataType], names: List[Optional[str]]
+ self,
+ index_name: Optional[str],
+ index_dtype: Dtype,
+ data_dtypes: List[Dtype],
+ data_names: List[Optional[str]],
+ spark_types: List[types.DataType],
):
from pyspark.pandas.internal import InternalField
from pyspark.pandas.utils import name_like_string
+ names = [index_name] + data_names if index_dtype is not None else data_names
+ dtypes = [index_dtype] + data_dtypes if index_dtype is not None else data_dtypes
+
+ self.index_name = index_name
+ self.index_dtype = index_dtype
+ self.data_dtypes = data_dtypes
+ self.data_names = data_names
Review comment:
I guess we should manage `index_field` and `data_fields` instead?
We can do it later, though.
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
+
+ Typing data columns only:
+
+ >>> pd.DataFrame[float, float]
Review comment:
I'm wondering why we are using `pd` instead of `ps`?
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
Review comment:
What's this example for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916835348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor edited a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
dgd-contributor edited a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917655939
Just curious, with your example
``` python
>>> def transform(pdf) -> pd.DataFrame[int, int]:
... pdf['A'] = pdf.id + 1
... return pdf
...
>>> ks.range(5).koalas.apply_batch(transform)
```
Could you please explain how the index type hint improve the performance? I read [default index](https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type) and [Specify the index column in conversion from Spark DataFrame to Koalas DataFrame](https://koalas.readthedocs.io/en/latest/user_guide/best_practices.html#specify-the-index-column-in-conversion-from-spark-dataframe-to-koalas-dataframe) but I have not understood. Thank you so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707859153
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
Review comment:
ah, got it. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707855604
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
Review comment:
This is actually for below:
```
pd.DataFrame[(pdf.index.name, pdf.index.dtype), zip(pdf.columns, pdf.dtypes)]
```
this syntax was added in case users have an easier way to create dtypes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916806504
**[Test build #143144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143144/testReport)** for PR 33954 at commit [`498aca3`](https://github.com/apache/spark/commit/498aca3cff7d8d9216fed307f3ace741954c26af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916730282
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916721855
**[Test build #143141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143141/testReport)** for PR 33954 at commit [`6675d40`](https://github.com/apache/spark/commit/6675d404e036dd32537169b081904684209cf9cc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class IndexNameTypeHolder(object):`
* ` new_class = type(\"NameType\", (NameTypeHolder,), `
* ` new_class = param.type if isinstance(param, np.dtype) else param`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917655939
Just curious, with your example
``` python
>>> def transform(pdf) -> pd.DataFrame[int, int]:
... pdf['A'] = pdf.id + 1
... return pdf
...
>>> ks.range(5).koalas.apply_batch(transform)
```
Could you please explain how the index type hint improve performance? I read [default index](https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type) and [Specify the index column in conversion from Spark DataFrame to Koalas DataFrame](https://koalas.readthedocs.io/en/latest/user_guide/best_practices.html#specify-the-index-column-in-conversion-from-spark-dataframe-to-koalas-dataframe) but I have not understood. Thank you so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916703270
cc @ueshin @xinrong-databricks @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918774029
**[Test build #143233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143233/testReport)** for PR 33954 at commit [`23c73c4`](https://github.com/apache/spark/commit/23c73c446f6ed569e6d90e4924291a0163a99713).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916824714
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47648/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916705819
**[Test build #143141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143141/testReport)** for PR 33954 at commit [`6675d40`](https://github.com/apache/spark/commit/6675d404e036dd32537169b081904684209cf9cc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916705819
**[Test build #143141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143141/testReport)** for PR 33954 at commit [`6675d40`](https://github.com/apache/spark/commit/6675d404e036dd32537169b081904684209cf9cc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916835348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r706067769
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
Review comment:
Multiindex will be done separately at SPARK-36711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707863591
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -84,11 +87,24 @@ def __repr__(self) -> str:
class DataFrameType(object):
def __init__(
- self, dtypes: List[Dtype], spark_types: List[types.DataType], names: List[Optional[str]]
+ self,
+ index_name: Optional[str],
+ index_dtype: Dtype,
+ data_dtypes: List[Dtype],
+ data_names: List[Optional[str]],
+ spark_types: List[types.DataType],
):
from pyspark.pandas.internal import InternalField
from pyspark.pandas.utils import name_like_string
+ names = [index_name] + data_names if index_dtype is not None else data_names
+ dtypes = [index_dtype] + data_dtypes if index_dtype is not None else data_dtypes
+
+ self.index_name = index_name
+ self.index_dtype = index_dtype
+ self.data_dtypes = data_dtypes
+ self.data_names = data_names
Review comment:
Oh yeah.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-919620972
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916806504
**[Test build #143144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143144/testReport)** for PR 33954 at commit [`498aca3`](https://github.com/apache/spark/commit/498aca3cff7d8d9216fed307f3ace741954c26af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918788416
**[Test build #143233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143233/testReport)** for PR 33954 at commit [`23c73c4`](https://github.com/apache/spark/commit/23c73c446f6ed569e6d90e4924291a0163a99713).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33954:
URL: https://github.com/apache/spark/pull/33954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-919620141
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917742920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-917742920
When the index type is not specified, the default index would be attached by default for the output DataFrame. Using default index usually brings some performance penalty. See the pseudo code below.
Assume we're using distributed sequence index (https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type), and we perform an panda API such as:
```python
df.apply(func)
```
Internally the Spark operations as below would be performed (pseudo code)
**Without index type hint:**
```python
spark_df = df.to_spark()
# compute date part
spark_df.select(udf(func, "data column types without index"))
# compute index part
spark.createDataFrame(spark_df.rdd.zipWithIndex().map(lambda p: p[1], p[0])) # attach index
```
**With index type hint:**
```python
spark_df = df.to_spark()
spark_df.select(udf(func, "column types with both data and index"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916734880
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916820045
**[Test build #143144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143144/testReport)** for PR 33954 at commit [`498aca3`](https://github.com/apache/spark/commit/498aca3cff7d8d9216fed307f3ace741954c26af).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707855277
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
+
+ Typing data columns only:
+
+ >>> pd.DataFrame[float, float]
Review comment:
maybe I should use ps to make it less confusing in the example.
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
+
+ Typing data columns only:
+
+ >>> pd.DataFrame[float, float]
Review comment:
maybe I should use ps to make it less confusing in the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-916731726
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918792682
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33954:
URL: https://github.com/apache/spark/pull/33954#discussion_r707829443
##########
File path: python/pyspark/pandas/typedef/typehints.py
##########
@@ -538,6 +634,165 @@ def infer_return_type(f: Callable) -> Union[SeriesType, DataFrameType, ScalarTyp
return ScalarType(*types)
+# TODO: once pandas exposes a typing module like numpy.typing, we should deprecate
+# this logic and migrate to it with implementing the typing module in pandas API on Spark.
+
+
+def create_type_for_series_type(param: Any) -> Type[SeriesType]:
+ """
+ Supported syntax:
+
+ >>> str(pd.Series[float]).endswith("SeriesType[float]")
+ True
+ """
+ from pyspark.pandas.typedef import NameTypeHolder
+
+ if isinstance(param, ExtensionDtype):
+ new_class = type("NameType", (NameTypeHolder,), {}) # type: Type[NameTypeHolder]
+ new_class.tpe = param
+ else:
+ new_class = param.type if isinstance(param, np.dtype) else param
+
+ return SeriesType[new_class] # type: ignore
+
+
+# TODO: Remove this variadic-generic hack by tuple once ww drop Python up to 3.9.
+# See also PEP 646. One problem is that pandas doesn't inherits Generic[T]
+# so we might have to leave this hack only for monkey-patching pandas DataFrame.
+def create_tuple_for_frame_type(params: Any) -> object:
+ """
+ This is a workaround to support variadic generic in DataFrame.
+
+ See https://github.com/python/typing/issues/193
+ we always wraps the given type hints by a tuple to mimic the variadic generic.
+
+ Supported syntax:
+
+ >>> import pandas as pd
+ >>> pdf = pd.DataFrame({'a': range(1)})
+
+ Typing data columns only:
+
+ >>> pd.DataFrame[float, float]
Review comment:
maybe I should use ps to make it less confusing in the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33954: [SPARK-36709][PYTHON] Support new syntax for specifying index type and name in pandas API on Spark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33954:
URL: https://github.com/apache/spark/pull/33954#issuecomment-918795993
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org