You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/24 10:01:34 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
HyukjinKwon opened a new pull request #28327:
URL: https://github.com/apache/spark/pull/28327
### What changes were proposed in this pull request?
This PR proposes to use a different approach instead of breaking it per Micheal's rubric added at https://spark.apache.org/versioning-policy.html. It deprecates the behaviour for now. It will be gradually removed in the future releases.
After this change,
```python
import warnings
warnings.simplefilter("always")
from pyspark.sql.functions import *
df = spark.range(2)
map_col = create_map(lit(0), lit(100), lit(1), lit(200))
df.withColumn("mapped", map_col.getItem(col('id'))).show()
```
```
/.../python/pyspark/sql/column.py:311: DeprecationWarning: A column as 'key' in getItem is
deprecated as of Spark 3.0, and will not be supported in the future release. Use `column[key]`
or `column.key` syntax instead.
DeprecationWarning)
...
```
### Why are the changes needed?
To prevent the radical behaviour change after the amended versioning policy.
### Does this PR introduce any user-facing change?
Yes, it will show the deprecated warning message.
### How was this patch tested?
Manually tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619709672
**[Test build #121872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619738372
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618923314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618922861
**[Test build #121749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121749/testReport)** for PR 28327 at commit [`eb2d788`](https://github.com/apache/spark/commit/eb2d788da45394e4b313a7b82b0eed45219b693e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618941641
**[Test build #121749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121749/testReport)** for PR 28327 at commit [`eb2d788`](https://github.com/apache/spark/commit/eb2d788da45394e4b313a7b82b0eed45219b693e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619726343
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r415487078
##########
File path: python/pyspark/sql/tests/test_column.py
##########
@@ -87,24 +85,14 @@ def test_column_operators(self):
"Cannot apply 'in' operator against a column",
lambda: 1 in cs)
- def test_column_apply(self):
+ def test_column_accessor(self):
from pyspark.sql.functions import col
self.assertIsInstance(col("foo")[1:3], Column)
self.assertIsInstance(col("foo")[0], Column)
self.assertIsInstance(col("foo")["bar"], Column)
self.assertRaises(ValueError, lambda: col("foo")[0:10:2])
- def test_column_getitem(self):
- from pyspark.sql.functions import col, create_map, lit
-
- map_col = create_map(lit(0), lit(100), lit(1), lit(200))
- self.assertRaisesRegexp(
- Py4JJavaError,
- "Unsupported literal type class org.apache.spark.sql.Column id",
Review comment:
It can be a valid test case yes. But let's just go simple. We should just target to print out a warning message against this specific case per the JIRA, which is tricky to test out, and I manually tested it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618923314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619710075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619687475
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r414644438
##########
File path: python/pyspark/sql/column.py
##########
@@ -296,12 +299,17 @@ def getItem(self, key):
+----+------+
| 1| value|
+----+------+
-
- .. versionchanged:: 3.0
- If `key` is a `Column` object, the indexing operator should be used instead.
- For example, `map_col.getItem(col('id'))` should be replaced with `map_col[col('id')]`.
"""
- return _bin_op("getItem")(self, key)
+ if isinstance(key, Column) and not is_instance_of(
+ SparkContext._gateway,
+ key._jc.expr(),
+ "org.apache.spark.sql.catalyst.expressions.Literal"):
Review comment:
So we allow Column of Literal?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619687165
**[Test build #121863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121863/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r415487078
##########
File path: python/pyspark/sql/tests/test_column.py
##########
@@ -87,24 +85,14 @@ def test_column_operators(self):
"Cannot apply 'in' operator against a column",
lambda: 1 in cs)
- def test_column_apply(self):
+ def test_column_accessor(self):
from pyspark.sql.functions import col
self.assertIsInstance(col("foo")[1:3], Column)
self.assertIsInstance(col("foo")[0], Column)
self.assertIsInstance(col("foo")["bar"], Column)
self.assertRaises(ValueError, lambda: col("foo")[0:10:2])
- def test_column_getitem(self):
- from pyspark.sql.functions import col, create_map, lit
-
- map_col = create_map(lit(0), lit(100), lit(1), lit(200))
- self.assertRaisesRegexp(
- Py4JJavaError,
- "Unsupported literal type class org.apache.spark.sql.Column id",
Review comment:
It can be a valid test case yes. But let's just go simple. We should just target to print out a warning message against this specific case, which is tricky to test out, and I manually tested it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619697549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618922861
**[Test build #121749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121749/testReport)** for PR 28327 at commit [`eb2d788`](https://github.com/apache/spark/commit/eb2d788da45394e4b313a7b82b0eed45219b693e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619706792
**[Test build #121868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121868/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619707004
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618920951
cc @imback82 and @viirya can you take a look please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619707526
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619707004
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619726343
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619697397
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r415484296
##########
File path: python/pyspark/sql/tests/test_column.py
##########
@@ -87,24 +85,14 @@ def test_column_operators(self):
"Cannot apply 'in' operator against a column",
lambda: 1 in cs)
- def test_column_apply(self):
+ def test_column_accessor(self):
from pyspark.sql.functions import col
self.assertIsInstance(col("foo")[1:3], Column)
self.assertIsInstance(col("foo")[0], Column)
self.assertIsInstance(col("foo")["bar"], Column)
self.assertRaises(ValueError, lambda: col("foo")[0:10:2])
- def test_column_getitem(self):
- from pyspark.sql.functions import col, create_map, lit
-
- map_col = create_map(lit(0), lit(100), lit(1), lit(200))
- self.assertRaisesRegexp(
- Py4JJavaError,
- "Unsupported literal type class org.apache.spark.sql.Column id",
Review comment:
do we still need this test via the accessor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
imback82 commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r415484818
##########
File path: python/pyspark/sql/tests/test_column.py
##########
@@ -87,24 +85,14 @@ def test_column_operators(self):
"Cannot apply 'in' operator against a column",
lambda: 1 in cs)
- def test_column_apply(self):
+ def test_column_accessor(self):
from pyspark.sql.functions import col
self.assertIsInstance(col("foo")[1:3], Column)
self.assertIsInstance(col("foo")[0], Column)
self.assertIsInstance(col("foo")["bar"], Column)
self.assertRaises(ValueError, lambda: col("foo")[0:10:2])
- def test_column_getitem(self):
- from pyspark.sql.functions import col, create_map, lit
-
- map_col = create_map(lit(0), lit(100), lit(1), lit(200))
- self.assertRaisesRegexp(
- Py4JJavaError,
- "Unsupported literal type class org.apache.spark.sql.Column id",
Review comment:
I mean, this is still a valid test case, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619698750
**[Test build #121868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121868/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619697397
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28327:
URL: https://github.com/apache/spark/pull/28327#discussion_r415204996
##########
File path: python/pyspark/sql/column.py
##########
@@ -296,12 +299,17 @@ def getItem(self, key):
+----+------+
| 1| value|
+----+------+
-
- .. versionchanged:: 3.0
- If `key` is a `Column` object, the indexing operator should be used instead.
- For example, `map_col.getItem(col('id'))` should be replaced with `map_col[col('id')]`.
"""
- return _bin_op("getItem")(self, key)
+ if isinstance(key, Column) and not is_instance_of(
+ SparkContext._gateway,
+ key._jc.expr(),
+ "org.apache.spark.sql.catalyst.expressions.Literal"):
Review comment:
That was my current take. It was because `getField` support a literal only at this moment as well (whereas Scala side only support `String`). I took a look for `functions.py`, and seems it's more consistent to just let it take string only, and no `Column` instance. Thanks for pointing this one ..
I will update this PR accordingly later tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619725648
**[Test build #121872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618942239
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619698750
**[Test build #121868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121868/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619707013
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121868/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619710075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619687475
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619697223
**[Test build #121863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121863/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619687165
**[Test build #121863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121863/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619709672
**[Test build #121872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-619699082
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28327:
URL: https://github.com/apache/spark/pull/28327#issuecomment-618942239
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org