You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/07 23:33:15 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
xinrong-databricks opened a new pull request #33929:
URL: https://github.com/apache/spark/pull/33929
### What changes were proposed in this pull request?
Support dropping rows of a single-indexed DataFrame.
### Why are the changes needed?
To increase pandas API coverage.
### Does this PR introduce _any_ user-facing change?
Yes, dropping rows of a single-indexed DataFrame is supported now.
```py
>>> df = ps.DataFrame(np.arange(12).reshape(3, 4), columns=['A', 'B', 'C', 'D'])
>>> df
A B C D
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
### From ###
>>> df.drop([0, 1])
Traceback (most recent call last):
...
KeyError: [(0,), (1,)]
>>> df.drop([0, 1], axis=0)
Traceback (most recent call last):
...
NotImplementedError: Drop currently only works for axis=1
>>> df.drop(1)
Traceback (most recent call last):
...
KeyError: [(1,)]
>>> df.drop(index=1)
...
TypeError: drop() got an unexpected keyword argument 'index'
### To ###
>>> df.drop([0, 1])
A B C D
2 8 9 10 11
>>> df.drop([0, 1], axis=0)
A B C D
2 8 9 10 11
>>> df.drop(1)
A B C D
0 0 1 2 3
2 8 9 10 11
>>> df.drop(index=1)
A B C D
0 0 1 2 3
2 8 9 10 11
```
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916370650
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47629/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916341684
**[Test build #143125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143125/testReport)** for PR 33929 at commit [`070a31e`](https://github.com/apache/spark/commit/070a31e94f94601b5eea953c063957fe22c6bb88).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r704654955
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1287,8 +1278,42 @@ def test_drop(self):
pdf = pd.DataFrame({10: [1, 2], 20: [3, 4], 30: [5, 6]}, index=np.random.rand(2))
psdf = ps.from_pandas(pdf)
- self.assert_eq(psdf.drop(10), pdf.drop(10, axis=1))
- self.assert_eq(psdf.drop([20, 30]), pdf.drop([20, 30], axis=1))
+ self.assert_eq(psdf.drop(10, axis=1), pdf.drop(10, axis=1))
+ self.assert_eq(psdf.drop([20, 30], axis=1), pdf.drop([20, 30], axis=1))
+
+ #
+ # Drop rows by index
+ #
+
+ pdf = pd.DataFrame({"X": [1, 2, 3], "Y": [4, 5, 6], "Z": [7, 8, 9]}, index=["A", "B", "C"])
+ psdf = ps.from_pandas(pdf)
+
+ # Given labels (and axis = 0)
+ self.assert_eq(psdf.drop(labels="A", axis=0), pdf.drop(labels="A", axis=0))
+ self.assert_eq(psdf.drop(labels="A"), pdf.drop(labels="A"))
+ self.assert_eq(psdf.drop(labels=["A", "C"], axis=0), pdf.drop(labels=["A", "C"], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=["A", "B", "C"], axis=0), pdf.drop(labels=["A", "B", "C"], axis=0)
+ )
+
+ # Given index
+ self.assert_eq(psdf.drop(index="A"), pdf.drop(index="A"))
+ self.assert_eq(psdf.drop(index=["A", "C"]), pdf.drop(index=["A", "C"]))
+ self.assert_eq(psdf.drop(index=["A", "B", "C"]), pdf.drop(index=["A", "B", "C"]))
+
+ # Non-string names
+ pdf.index = [10, 20, 30]
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(psdf.drop(labels=10, axis=0), pdf.drop(labels=10, axis=0))
+ self.assert_eq(psdf.drop(labels=[10, 30], axis=0), pdf.drop(labels=[10, 30], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=[10, 20, 30], axis=0), pdf.drop(labels=[10, 20, 30], axis=0)
+ )
+
+ # MultiIndex
+ pdf.index = pd.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])
+ psdf = ps.from_pandas(pdf)
+ self.assertRaises(NotImplementedError, lambda: psdf.drop(labels=[("a", "x")]))
Review comment:
Good point! Let me support that in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918661894
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47728/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921338365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919420667
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143271/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708506569
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
+ how="anti",
+ )
+ internal = internal.with_new_sdf(joined_sdf)
Review comment:
Why is `index_sdf_col` in the joined frame?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919423022
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916297511
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r703961645
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1287,8 +1278,42 @@ def test_drop(self):
pdf = pd.DataFrame({10: [1, 2], 20: [3, 4], 30: [5, 6]}, index=np.random.rand(2))
psdf = ps.from_pandas(pdf)
- self.assert_eq(psdf.drop(10), pdf.drop(10, axis=1))
- self.assert_eq(psdf.drop([20, 30]), pdf.drop([20, 30], axis=1))
+ self.assert_eq(psdf.drop(10, axis=1), pdf.drop(10, axis=1))
+ self.assert_eq(psdf.drop([20, 30], axis=1), pdf.drop([20, 30], axis=1))
+
+ #
+ # Drop rows by index
+ #
+
+ pdf = pd.DataFrame({"X": [1, 2, 3], "Y": [4, 5, 6], "Z": [7, 8, 9]}, index=["A", "B", "C"])
+ psdf = ps.from_pandas(pdf)
+
+ # Given labels (and axis = 0)
+ self.assert_eq(psdf.drop(labels="A", axis=0), pdf.drop(labels="A", axis=0))
+ self.assert_eq(psdf.drop(labels="A"), pdf.drop(labels="A"))
+ self.assert_eq(psdf.drop(labels=["A", "C"], axis=0), pdf.drop(labels=["A", "C"], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=["A", "B", "C"], axis=0), pdf.drop(labels=["A", "B", "C"], axis=0)
+ )
+
+ # Given index
+ self.assert_eq(psdf.drop(index="A"), pdf.drop(index="A"))
+ self.assert_eq(psdf.drop(index=["A", "C"]), pdf.drop(index=["A", "C"]))
+ self.assert_eq(psdf.drop(index=["A", "B", "C"]), pdf.drop(index=["A", "B", "C"]))
+
+ # Non-string names
+ pdf.index = [10, 20, 30]
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(psdf.drop(labels=10, axis=0), pdf.drop(labels=10, axis=0))
+ self.assert_eq(psdf.drop(labels=[10, 30], axis=0), pdf.drop(labels=[10, 30], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=[10, 20, 30], axis=0), pdf.drop(labels=[10, 20, 30], axis=0)
+ )
+
+ # MultiIndex
+ pdf.index = pd.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])
+ psdf = ps.from_pandas(pdf)
+ self.assertRaises(NotImplementedError, lambda: psdf.drop(labels=[("a", "x")]))
Review comment:
Should we support drop columns and row in a single time ? Like
```python
>>> pdf = pd.DataFrame({"X": [1, 2, 3], "Y": [4, 5, 6], "Z": [7, 8, 9]}, index=["A", "B", "C"])
>>> pdf.drop(columns="X", index="A")
Y Z
B 5 8
C 6 9
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916537518
**[Test build #143132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143132/testReport)** for PR 33929 at commit [`72dd09d`](https://github.com/apache/spark/commit/72dd09d797252123e642af140232b4894791ed1a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916430464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143126/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921343453
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916358907
**[Test build #143125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143125/testReport)** for PR 33929 at commit [`070a31e`](https://github.com/apache/spark/commit/070a31e94f94601b5eea953c063957fe22c6bb88).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r709674753
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6631,23 +6631,25 @@ def droplevel(
def drop(
self,
labels: Optional[Union[Name, List[Name]]] = None,
- axis: Axis = 1,
+ axis: Optional[Axis] = 0,
Review comment:
Actually this is a behavior change between 3.2 and 3.3.
Shall we add `versionchanged` directive with explanation in the docstring, and document this in a migration guide?
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,96 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= ps.get_option("compute.isin_limit"):
+ self_index_type = self.index.to_series().spark.data_type
Review comment:
nit: `self.index.spark.data_type` should be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r704895060
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
I am not sure but I think loop over the indexes may be not efficient. What do you think about create ```sdf``` from index and make an anti join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916561864
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917261142
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143159/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r707775701
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
Review comment:
`80` should be replaced by 'compute.isin_limit' after https://issues.apache.org/jira/browse/SPARK-33383 merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919413419
**[Test build #143271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143271/testReport)** for PR 33929 at commit [`765b4f8`](https://github.com/apache/spark/commit/765b4f8d4046bf2bd3c3222fd4f04386851cfeac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r704922631
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6631,23 +6631,25 @@ def droplevel(
def drop(
self,
labels: Optional[Union[Name, List[Name]]] = None,
- axis: Axis = 1,
+ axis: Optional[Axis] = 0,
Review comment:
Could you review the code which use ps.DataFrame.drop() when change default axis? Like
https://github.com/apache/spark/blob/b445cfd73fe659d09922ddd6a80ea7edbf6c26fc/python/pyspark/pandas/frame.py#L2865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916549380
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47636/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915473058
**[Test build #143091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143091/testReport)** for PR 33929 at commit [`b445cfd`](https://github.com/apache/spark/commit/b445cfd73fe659d09922ddd6a80ea7edbf6c26fc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919393172
**[Test build #143271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143271/testReport)** for PR 33929 at commit [`765b4f8`](https://github.com/apache/spark/commit/765b4f8d4046bf2bd3c3222fd4f04386851cfeac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r710540726
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6631,23 +6631,25 @@ def droplevel(
def drop(
self,
labels: Optional[Union[Name, List[Name]]] = None,
- axis: Axis = 1,
+ axis: Optional[Axis] = 0,
Review comment:
Makes sense! Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916419321
**[Test build #143126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143126/testReport)** for PR 33929 at commit [`f8192f0`](https://github.com/apache/spark/commit/f8192f025bdcd9f1a64fe7b0226600b17e3b99eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919422953
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916549380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r706500814
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,75 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
-
- keep_columns_and_labels = [
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+ self_index_column = self.index.to_series()
+ self_index_type = self_index_column.spark.data_type
+ cond = ~self_index_column.spark.column.isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ filtered_sdf = internal.spark_frame.filter(cond)
+ internal = internal.with_new_sdf(filtered_sdf)
Review comment:
The refactored code looks cleaner.
Empty list input has been considered and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705822599
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
Could you take a look at this issue https://issues.apache.org/jira/browse/SPARK-33383
I think the solution is depend on this size of ```index``` and I do not know the size of index used by user often
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918665514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708508501
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
Review comment:
Good point, follow-up PRs may be filed if customers request. Otherwise, I guess `list` is good for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919420667
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143271/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921333335
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918637378
**[Test build #143225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143225/testReport)** for PR 33929 at commit [`a34e20a`](https://github.com/apache/spark/commit/a34e20aff4212110fb8ee5a82585e968f8748721).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705710878
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,78 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
-
- keep_columns_and_labels = [
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if internal.index_level == 1:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ self_sdf = internal.spark_frame
+ joined_sdf = self_sdf.join(
+ other=F.broadcast(index_sdf),
+ on=scol_for(self_sdf, SPARK_INDEX_NAME_FORMAT(0))
Review comment:
`internal.index_spark_columns[0]` instead of `scol_for(sdf, SPARK_INDEX_NAME_FORMAT(0))`?
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,78 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
Review comment:
nit: I'd put this to line 6731.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916284174
**[Test build #143122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143122/testReport)** for PR 33929 at commit [`e420c52`](https://github.com/apache/spark/commit/e420c5271aa528a1efe603ecc063c19fc6d38889).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916489967
**[Test build #143130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143130/testReport)** for PR 33929 at commit [`5b6faac`](https://github.com/apache/spark/commit/5b6faac5cc7d01ba61f92e357ffc1fa7c13ce394).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915510715
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916263876
**[Test build #143122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143122/testReport)** for PR 33929 at commit [`e420c52`](https://github.com/apache/spark/commit/e420c5271aa528a1efe603ecc063c19fc6d38889).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914753817
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47565/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914694962
**[Test build #143062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143062/testReport)** for PR 33929 at commit [`7bc174e`](https://github.com/apache/spark/commit/7bc174ea5b835d2baf234f3652c6a34e845bc9c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705800784
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
Good idea! It saves creating a `sdf` for joining.
How do you think about that @dgd-contributor ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #33929:
URL: https://github.com/apache/spark/pull/33929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916537518
**[Test build #143132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143132/testReport)** for PR 33929 at commit [`72dd09d`](https://github.com/apache/spark/commit/72dd09d797252123e642af140232b4894791ed1a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916442605
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47630/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705803888
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,75 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
-
- keep_columns_and_labels = [
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+ self_index_column = self.index.to_series()
+ self_index_type = self_index_column.spark.data_type
+ cond = ~self_index_column.spark.column.isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ filtered_sdf = internal.spark_frame.filter(cond)
+ internal = internal.with_new_sdf(filtered_sdf)
Review comment:
btw, what if the `index` is an empty list?
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,75 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
-
- keep_columns_and_labels = [
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+ self_index_column = self.index.to_series()
+ self_index_type = self_index_column.spark.data_type
+ cond = ~self_index_column.spark.column.isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ filtered_sdf = internal.spark_frame.filter(cond)
+ internal = internal.with_new_sdf(filtered_sdf)
Review comment:
How about:
```py
cond = ~internal.index_spark_columns[0].isin( ... )
internal = internal.with_filter(cond)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916341684
**[Test build #143125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143125/testReport)** for PR 33929 at commit [`070a31e`](https://github.com/apache/spark/commit/070a31e94f94601b5eea953c063957fe22c6bb88).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916298869
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916483363
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708489617
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
Review comment:
May I ask how shall we check? `verify_temp_column_name` requires the frame exist. Since we are creating a new frame, the column name must not exist beforehand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r709456794
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
+ how="anti",
+ )
+ internal = internal.with_new_sdf(joined_sdf)
Review comment:
ah, my bad. As it's an anti-join, the column `index_sdf_col` won't appear.
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
Review comment:
I meant to check it with `internal.spark_frame`, but we don't need it as it's an anti-join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921338363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916430464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143126/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917282445
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r703919742
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6631,23 +6631,25 @@ def droplevel(
def drop(
self,
labels: Optional[Union[Name, List[Name]]] = None,
- axis: Axis = 1,
+ axis: Optional[Axis] = 0,
Review comment:
Set axis to `0` to follow pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r707775369
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
https://github.com/apache/spark/pull/33982 is created for avoiding the issue https://issues.apache.org/jira/browse/SPARK-33383.
Thank you both!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916298918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919393172
**[Test build #143271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143271/testReport)** for PR 33929 at commit [`765b4f8`](https://github.com/apache/spark/commit/765b4f8d4046bf2bd3c3222fd4f04386851cfeac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916298918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705673878
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
That really makes sense. Let me refactor that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916485528
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r704826279
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1287,8 +1278,42 @@ def test_drop(self):
pdf = pd.DataFrame({10: [1, 2], 20: [3, 4], 30: [5, 6]}, index=np.random.rand(2))
psdf = ps.from_pandas(pdf)
- self.assert_eq(psdf.drop(10), pdf.drop(10, axis=1))
- self.assert_eq(psdf.drop([20, 30]), pdf.drop([20, 30], axis=1))
+ self.assert_eq(psdf.drop(10, axis=1), pdf.drop(10, axis=1))
+ self.assert_eq(psdf.drop([20, 30], axis=1), pdf.drop([20, 30], axis=1))
+
+ #
+ # Drop rows by index
+ #
+
+ pdf = pd.DataFrame({"X": [1, 2, 3], "Y": [4, 5, 6], "Z": [7, 8, 9]}, index=["A", "B", "C"])
+ psdf = ps.from_pandas(pdf)
+
+ # Given labels (and axis = 0)
+ self.assert_eq(psdf.drop(labels="A", axis=0), pdf.drop(labels="A", axis=0))
+ self.assert_eq(psdf.drop(labels="A"), pdf.drop(labels="A"))
+ self.assert_eq(psdf.drop(labels=["A", "C"], axis=0), pdf.drop(labels=["A", "C"], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=["A", "B", "C"], axis=0), pdf.drop(labels=["A", "B", "C"], axis=0)
+ )
+
+ # Given index
+ self.assert_eq(psdf.drop(index="A"), pdf.drop(index="A"))
+ self.assert_eq(psdf.drop(index=["A", "C"]), pdf.drop(index=["A", "C"]))
+ self.assert_eq(psdf.drop(index=["A", "B", "C"]), pdf.drop(index=["A", "B", "C"]))
+
+ # Non-string names
+ pdf.index = [10, 20, 30]
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(psdf.drop(labels=10, axis=0), pdf.drop(labels=10, axis=0))
+ self.assert_eq(psdf.drop(labels=[10, 30], axis=0), pdf.drop(labels=[10, 30], axis=0))
+ self.assert_eq(
+ psdf.drop(labels=[10, 20, 30], axis=0), pdf.drop(labels=[10, 20, 30], axis=0)
+ )
+
+ # MultiIndex
+ pdf.index = pd.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])
+ psdf = ps.from_pandas(pdf)
+ self.assertRaises(NotImplementedError, lambda: psdf.drop(labels=[("a", "x")]))
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915502361
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916509063
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705795660
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
On second thoughts, I guess we should use `F.isin` instead of anti-join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918665610
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47728/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708508501
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
Review comment:
Good point, follow-up PRs may be filed if customers request. Otherwise, I guess `List` is good for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918665514
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705753805
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,78 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
-
- keep_columns_and_labels = [
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if internal.index_level == 1:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ self_sdf = internal.spark_frame
+ joined_sdf = self_sdf.join(
+ other=F.broadcast(index_sdf),
+ on=scol_for(self_sdf, SPARK_INDEX_NAME_FORMAT(0))
Review comment:
Makes sense! Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r707775701
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
Review comment:
`80` should be replaced by 'compute.isin_limit' after #33982 merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916377874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918652763
**[Test build #143225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143225/testReport)** for PR 33929 at commit [`a34e20a`](https://github.com/apache/spark/commit/a34e20aff4212110fb8ee5a82585e968f8748721).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921317357
**[Test build #143368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143368/testReport)** for PR 33929 at commit [`20e1e73`](https://github.com/apache/spark/commit/20e1e739a8dcc8909c30b2fd3a210f9ae3f7d7a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917271807
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917261142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914715605
**[Test build #143062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143062/testReport)** for PR 33929 at commit [`7bc174e`](https://github.com/apache/spark/commit/7bc174ea5b835d2baf234f3652c6a34e845bc9c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921330508
**[Test build #143368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143368/testReport)** for PR 33929 at commit [`20e1e73`](https://github.com/apache/spark/commit/20e1e739a8dcc8909c30b2fd3a210f9ae3f7d7a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921325320
**[Test build #143370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143370/testReport)** for PR 33929 at commit [`2419e61`](https://github.com/apache/spark/commit/2419e6145a0d2f98ccfc080ee777f9c6cb20e1a8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705707889
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,78 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
Review comment:
nit: I'd put this to line 6731.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-923348117
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921339608
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915489978
**[Test build #143091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143091/testReport)** for PR 33929 at commit [`b445cfd`](https://github.com/apache/spark/commit/b445cfd73fe659d09922ddd6a80ea7edbf6c26fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916544855
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47636/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914725140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916469757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916263876
**[Test build #143122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143122/testReport)** for PR 33929 at commit [`e420c52`](https://github.com/apache/spark/commit/e420c5271aa528a1efe603ecc063c19fc6d38889).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916452544
**[Test build #143128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143128/testReport)** for PR 33929 at commit [`f06dd0c`](https://github.com/apache/spark/commit/f06dd0cb95d80e2ec6cf604e31091d78ff880c59).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916509876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917252813
**[Test build #143159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143159/testReport)** for PR 33929 at commit [`4731a9a`](https://github.com/apache/spark/commit/4731a9a234cf824e30968e403723112ac0196e21).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916451948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47630/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705826338
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,50 +6706,71 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
+ internal = self._internal
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
- cols, labels = zip(
- *(
- (column, label)
- for column, label in zip(
- self._internal.data_spark_column_names, self._internal.column_labels
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ index_scols = internal.index_spark_columns
+ if len(index_scols) == 1:
+ col = None
+ for label in index:
+ if col is None:
+ col = index_scols[0] != SF.lit(label)
Review comment:
That's a good point, but if the index is small, anti-join is too expensive.
Maybe we can make the threshold configurable with `ps.option` and switch to use `isin` or anti-join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r707776125
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
Review comment:
Test cases will be added to test this `else` clause after https://github.com/apache/spark/pull/33982 merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708508501
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
Review comment:
Good point, I'd like to keep it as a `List` to be consistent with existing parameters. I may file a follow-up PR for making all parameters list-like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-919423022
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918665627
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47728/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915490374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916419321
**[Test build #143126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143126/testReport)** for PR 33929 at commit [`f8192f0`](https://github.com/apache/spark/commit/f8192f025bdcd9f1a64fe7b0226600b17e3b99eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916550970
**[Test build #143132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143132/testReport)** for PR 33929 at commit [`72dd09d`](https://github.com/apache/spark/commit/72dd09d797252123e642af140232b4894791ed1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916469757
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914725140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916501091
**[Test build #143130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143130/testReport)** for PR 33929 at commit [`5b6faac`](https://github.com/apache/spark/commit/5b6faac5cc7d01ba61f92e357ffc1fa7c13ce394).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916297511
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916291434
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917252813
**[Test build #143159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143159/testReport)** for PR 33929 at commit [`4731a9a`](https://github.com/apache/spark/commit/4731a9a234cf824e30968e403723112ac0196e21).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915490374
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143091/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914777320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47565/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916509877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916430112
**[Test build #143126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143126/testReport)** for PR 33929 at commit [`f8192f0`](https://github.com/apache/spark/commit/f8192f025bdcd9f1a64fe7b0226600b17e3b99eb).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916451948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47630/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r707775397
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
Review comment:
Shall we make `80` configurable?
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
Review comment:
Shall we check the name is valid with `verify_temp_column_name`?
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
Review comment:
nit: `on=( ... )`
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
+ how="anti",
+ )
+ internal = internal.with_new_sdf(joined_sdf)
Review comment:
I guess we should drop the column `index_sdf_col`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708508501
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
Review comment:
I'd like to keep it as a `List` to be consistent with existing parameters. I may file a follow-up PR for making all parameters list-like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708506261
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916469459
**[Test build #143128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143128/testReport)** for PR 33929 at commit [`f06dd0c`](https://github.com/apache/spark/commit/f06dd0cb95d80e2ec6cf604e31091d78ff880c59).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r708335131
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
Review comment:
Should we support index types like ps.Series, ps.Index, np.ndarray ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r709490851
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
+ index_sdf = default_session().createDataFrame(
+ pd.DataFrame({index_sdf_col: index})
+ )
+ joined_sdf = internal.spark_frame.join(
+ other=F.broadcast(index_sdf),
+ on=internal.index_spark_columns[0]
+ == scol_for(index_sdf, index_sdf_col),
+ how="anti",
+ )
+ internal = internal.with_new_sdf(joined_sdf)
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r710557949
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,96 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= ps.get_option("compute.isin_limit"):
+ self_index_type = self.index.to_series().spark.data_type
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-918637378
**[Test build #143225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143225/testReport)** for PR 33929 at commit [`a34e20a`](https://github.com/apache/spark/commit/a34e20aff4212110fb8ee5a82585e968f8748721).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921343488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916489967
**[Test build #143130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143130/testReport)** for PR 33929 at commit [`5b6faac`](https://github.com/apache/spark/commit/5b6faac5cc7d01ba61f92e357ffc1fa7c13ce394).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916377876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916479251
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47632/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914777320
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47565/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917260918
**[Test build #143159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143159/testReport)** for PR 33929 at commit [`4731a9a`](https://github.com/apache/spark/commit/4731a9a234cf824e30968e403723112ac0196e21).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-917288238
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914933856
cc @ueshin @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r705673357
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6631,23 +6631,25 @@ def droplevel(
def drop(
self,
labels: Optional[Union[Name, List[Name]]] = None,
- axis: Axis = 1,
+ axis: Optional[Axis] = 0,
Review comment:
Good catch, the places using `drop` have been updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914725049
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47565/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916506502
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915508077
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921336473
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921337550
**[Test build #143370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143370/testReport)** for PR 33929 at commit [`2419e61`](https://github.com/apache/spark/commit/2419e6145a0d2f98ccfc080ee777f9c6cb20e1a8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921325320
**[Test build #143370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143370/testReport)** for PR 33929 at commit [`2419e61`](https://github.com/apache/spark/commit/2419e6145a0d2f98ccfc080ee777f9c6cb20e1a8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921317357
**[Test build #143368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143368/testReport)** for PR 33929 at commit [`20e1e73`](https://github.com/apache/spark/commit/20e1e739a8dcc8909c30b2fd3a210f9ae3f7d7a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916452544
**[Test build #143128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143128/testReport)** for PR 33929 at commit [`f06dd0c`](https://github.com/apache/spark/commit/f06dd0cb95d80e2ec6cf604e31091d78ff880c59).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33929:
URL: https://github.com/apache/spark/pull/33929#discussion_r709494513
##########
File path: python/pyspark/pandas/frame.py
##########
@@ -6695,53 +6706,94 @@ def drop(
x y z w
0 1 3 5 7
1 2 4 6 8
- >>> df.drop('a') # doctest: +NORMALIZE_WHITESPACE
+ >>> df.drop(labels='a', axis=1) # doctest: +NORMALIZE_WHITESPACE
b
z w
0 5 7
1 6 8
Notes
-----
- Currently only axis = 1 is supported in this function,
- axis = 0 is yet to be implemented.
+ Currently, dropping rows of a MultiIndex DataFrame is not supported yet.
"""
if labels is not None:
+ if index is not None or columns is not None:
+ raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
axis = validate_axis(axis)
if axis == 1:
- return self.drop(columns=labels)
- raise NotImplementedError("Drop currently only works for axis=1")
- elif columns is not None:
- if is_name_like_tuple(columns):
- columns = [columns]
- elif is_name_like_value(columns):
- columns = [(columns,)]
+ return self.drop(index=index, columns=labels)
else:
- columns = [col if is_name_like_tuple(col) else (col,) for col in columns]
- drop_column_labels = set(
- label
- for label in self._internal.column_labels
- for col in columns
- if label[: len(col)] == col
- )
- if len(drop_column_labels) == 0:
- raise KeyError(columns)
+ return self.drop(index=labels, columns=columns)
+ else:
+ if index is None and columns is None:
+ raise ValueError("Need to specify at least one of 'labels' or 'columns' or 'index'")
+
+ internal = self._internal
+ if index is not None:
+ if is_name_like_tuple(index) or is_name_like_value(index):
+ index = [index]
+
+ if len(index) > 0:
+ if internal.index_level == 1:
+ internal = internal.resolved_copy
+
+ if len(index) <= 80:
+ self_index_type = self.index.to_series().spark.data_type
+ cond = ~internal.index_spark_columns[0].isin(
+ [SF.lit(label).cast(self_index_type) for label in index]
+ )
+ internal = internal.with_filter(cond)
+ else:
+ index_sdf_col = "__index"
Review comment:
Got it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-921343488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916549355
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47636/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-916448313
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47630/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33929: [SPARK-36618] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-914694962
**[Test build #143062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143062/testReport)** for PR 33929 at commit [`7bc174e`](https://github.com/apache/spark/commit/7bc174ea5b835d2baf234f3652c6a34e845bc9c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33929: [SPARK-36618][PYTHON] Support dropping rows of a single-indexed DataFrame
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33929:
URL: https://github.com/apache/spark/pull/33929#issuecomment-915473058
**[Test build #143091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143091/testReport)** for PR 33929 at commit [`b445cfd`](https://github.com/apache/spark/commit/b445cfd73fe659d09922ddd6a80ea7edbf6c26fc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org