You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/16 14:51:04 UTC
[GitHub] [spark] zero323 opened a new pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
zero323 opened a new pull request #31207:
URL: https://github.com/apache/spark/pull/31207
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR extends `pyspark.sql.functions.lit` to support complex literals:
- `list` mapped to SQL `array`.
- `dict` mapped to SQL `map`.
- `tuple` mapped to `struct` (with support for capturing names from `namedtuple` and `Row`)
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently Python uses have to quite verbose constructs to create complex literals, i.e.
```python
from pyspark.sql.functions import array, lit
xs = [1, 2, 3]
array(*[lit(x) for x in xs])
```
```python
from pyspark.sql.functions import create_map, lit, map_from_arrays
from itertools import chain
kvs = {"a": 1, "b": 2}
create_map(*chain.from_iterable(
(lit(k), lit(v)) for k, v in kvs.items()
))
# or
map_from_arrays(
array(*[lit(k) for k in kvs.keys()]),
array(*[lit(v) for v in kvs.values()])
)
```
In contrast, such literals can be created in Scala using `typedLit`.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. It is merged users will be able to create complex literals directly with `lit`, i.e.
```python
lit({"a": 1, "b": 2})
lit(["foo", "bar"])
lit((1, ("a", True)))
```
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
New unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r567530903
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
+
+ If it is a collection, conversion will be applied recursively. In such case,
+ all stored values should be of compatible types.
+
+ The passed in object is returned directly if it is already a :class:`Column`.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
Review comment:
One thing I am a bit worried is that this will lazily check the type (with type coercion) unlike Scala side ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769209778
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566116398
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
Review comment:
Added a short description of the input parameters @LielBach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565030565
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
cc @viirya too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 edited a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 edited a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761621037
__Note__:
A cleaner way to implement would be to use `singledispatch` (see possible implementation [here](https://github.com/apache/spark/compare/master...zero323:SPARK-34136-dispatched?expand=1)).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769810438
**[Test build #134656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134656/testReport)** for PR 31207 at commit [`dade2b1`](https://github.com/apache/spark/commit/dade2b17152baf3f02553060bc8bbdbf96744756).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761586012
**[Test build #134142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134142/testReport)** for PR 31207 at commit [`b477f42`](https://github.com/apache/spark/commit/b477f42c15010e02b59cf34b41ef51fbb30404a7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769857880
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39244/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761584849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134141/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761584849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134141/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-768022562
Oops, I missed this Pr. Yeah, make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566624021
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
It seems fine, but we might be able to convert these exprs for complex types into literals in advance via `PythonSQLUtils`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-860129044
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761586012
**[Test build #134142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134142/testReport)** for PR 31207 at commit [`b477f42`](https://github.com/apache/spark/commit/b477f42c15010e02b59cf34b41ef51fbb30404a7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761621037
__Note__:
A cleaner way to implement it would be to use `singledispatch` (see possible implementation [here](https://github.com/apache/spark/compare/master...zero323:SPARK-34136-dispatched?expand=1)).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761587363
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38724/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-790606346
Any update?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565052739
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
Seems okay because there is constant folding optimization. So if they are foldable, they will be optimized as literals during query optimization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761596711
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38725/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769142839
**[Test build #134620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134620/testReport)** for PR 31207 at commit [`6198e0a`](https://github.com/apache/spark/commit/6198e0aca0d30d6bd6ba9f29a0fe60a62016caed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565027496
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
One thing that makes me worry is this isn't technically a literal ... although it's foldable, the results and optimization might be differently applied compared to Scala side .. also looks like Scala side doesn't support map and struct (due to both the limitation of typing and the fact that `Row` doesn't hold field names, etc.).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566568600
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
Looks fine, too. As @zero323 said above, `typedLit` supports the complex types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761594995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761576393
**[Test build #134141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134141/testReport)** for PR 31207 at commit [`d4ac8b4`](https://github.com/apache/spark/commit/d4ac8b411caece0979c5f15e6e0b6a17373669dd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LielBach commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
LielBach commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r564876599
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
Review comment:
Worth mentioning the newly supported types in the docstring
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565027496
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
One thing that makes me worry is this isn't technically a literal ... although it's foldable, the results and optimization might be differently applied compared to Scala side .. also looks like Scala side doesn't support map and struct (due to both the limitation of typing and the fact that `Row` doesn't hold field names, etc. like PySpark).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566820167
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
Review comment:
I am not sure if I understand what you mean. We already have matching sentence in the description block
https://github.com/apache/spark/blob/6198e0aca0d30d6bd6ba9f29a0fe60a62016caed/python/pyspark/sql/functions.py#L92
Could you clarify please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769878358
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39244/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565336841
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
> Also looks like Scala side doesn't support map and struct (due to both the limitation of typing and the fact that `Row` doesn't hold field names, etc.).
Actually, `typedLit` supports `Map` objects:
```scala
scala> typedLit(Map("a" -> 1, "b" -> 2))
res7: org.apache.spark.sql.Column = keys: [a,b], values: [1,2]
```
and `Products`
```scala
scala> case class FooBar(foo: Int, bar: String)
defined class FooBar
scala> typedLit(Foo(1, "a"))
res8: org.apache.spark.sql.Column = [1,a]
```
`Row` would be specific to PySpark, but since it is just a tuple, it seemed reasonable to handle it with names.
> Hm, okay but these will be literals after constant folding though.
The biggest difference compared to Scala is that it we handle mixed constants and column objects:
```python
>>> spark.createDataFrame([(1, 2)], ("a", "b")).select(lit({"a": col("a"), "b": col("b")})).show()
+----------------+
| map(a, a, b, b)|
+----------------+
|{a -> 1, b -> 2}|
+----------------+
```
So there is a case where `lit` results in a non-`Literal`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-768313838
Okie, I am good with this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769839484
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39244/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769108495
**[Test build #134620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134620/testReport)** for PR 31207 at commit [`6198e0a`](https://github.com/apache/spark/commit/6198e0aca0d30d6bd6ba9f29a0fe60a62016caed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566569587
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
Review comment:
"Creates a :class:`Column` of literal value."?
https://github.com/apache/spark/blob/72b7f8abfb60d0008f1f9bed94ce1c367a7d7cce/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L105
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761576393
**[Test build #134141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134141/testReport)** for PR 31207 at commit [`d4ac8b4`](https://github.com/apache/spark/commit/d4ac8b411caece0979c5f15e6e0b6a17373669dd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769842849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769168004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134620/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761582500
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38724/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761604736
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38725/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LielBach commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
LielBach commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r564876599
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
Review comment:
Worth mentioning the newly supported types in the docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769209778
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769168004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134620/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565336841
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
> Also looks like Scala side doesn't support map and struct (due to both the limitation of typing and the fact that `Row` doesn't hold field names, etc.).
Actually, `typedLit` supports `Map` objects:
```scala
scala> typedLit(Map("a" -> 1, "b" -> 2))
res7: org.apache.spark.sql.Column = keys: [a,b], values: [1,2]
```
and `Products`
```scala
scala> case class FooBar(foo: Int, bar: String)
defined class FooBar
scala> typedLit(FooBar(1, "a"))
res8: org.apache.spark.sql.Column = [1,a]
```
`Row` would be specific to PySpark, but since it is just a tuple, it seemed reasonable to handle it with names.
> Hm, okay but these will be literals after constant folding though.
The biggest difference compared to Scala is that it we handle mixed constants and column objects:
```python
>>> spark.createDataFrame([(1, 2)], ("a", "b")).select(lit({"a": col("a"), "b": col("b")})).show()
+----------------+
| map(a, a, b, b)|
+----------------+
|{a -> 1, b -> 2}|
+----------------+
```
So there is a case where `lit` results in a non-`Literal`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769108495
**[Test build #134620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134620/testReport)** for PR 31207 at commit [`6198e0a`](https://github.com/apache/spark/commit/6198e0aca0d30d6bd6ba9f29a0fe60a62016caed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566570144
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
+
+ If it is a collection, conversion will be applied recursively. In such case,
+ all stored values should be of compatible types.
+
+ I `col` is already a :class:`Column`, it will be returned unmodified.
Review comment:
ditto: "The passed in object is returned directly if it is already a :class:`Column`."
https://github.com/apache/spark/blob/72b7f8abfb60d0008f1f9bed94ce1c367a7d7cce/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L107
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769823276
**[Test build #134656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134656/testReport)** for PR 31207 at commit [`dade2b1`](https://github.com/apache/spark/commit/dade2b17152baf3f02553060bc8bbdbf96744756).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566568861
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
+
+ If it is a collection, conversion will be applied recursively. In such case,
+ all stored values should be of compatible types.
+
+ I `col` is already a :class:`Column`, it will be returned unmodified.
Review comment:
nit: how about `I` -> `If` and removing "already".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761589384
**[Test build #134142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134142/testReport)** for PR 31207 at commit [`b477f42`](https://github.com/apache/spark/commit/b477f42c15010e02b59cf34b41ef51fbb30404a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-768022562
Oops, I missed this Pr. Yeah, make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769842849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769188647
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LielBach commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
LielBach commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r564876599
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
Review comment:
Worth mentioning the supported types in the docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r567365177
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
+
+ If it is a collection, conversion will be applied recursively. In such case,
+ all stored values should be of compatible types.
+
+ The passed in object is returned directly if it is already a :class:`Column`.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
Review comment:
@zero323, can we implement these logics in Scala side, and we simply call `lit` here? I think that's more what @maropu meant.
I think the only special case we need to implement in Python side is the struct case because 1. `Row` class cannot be passed to JVM side and 2. both `list` and `tuple` are converted into `ArrayList` so JVM cannot tell the origin. We will probably have to add some comments on that case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565027496
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
One thing that makes me worry is this isn't technically a literal ... although it's foldable. The results and optimization might be differently applied compared to Scala side because they are not literals. For example, we have some optimization rules that only applies to literals here and there.
Also looks like Scala side doesn't support map and struct (due to both the limitation of typing and the fact that `Row` doesn't hold field names, etc.).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761579906
**[Test build #134141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134141/testReport)** for PR 31207 at commit [`d4ac8b4`](https://github.com/apache/spark/commit/d4ac8b411caece0979c5f15e6e0b6a17373669dd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r566568600
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
Looks fine, too. As @zero323 said above, `typedLit` supports the complex types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761594995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769810438
**[Test build #134656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134656/testReport)** for PR 31207 at commit [`dade2b1`](https://github.com/apache/spark/commit/dade2b17152baf3f02553060bc8bbdbf96744756).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #31207:
URL: https://github.com/apache/spark/pull/31207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r565030192
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,48 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
Examples
--------
>>> df.select(lit(5).alias('height')).withColumn('spark_user', lit(True)).take(1)
[Row(height=5, spark_user=True)]
- """
- return col if isinstance(col, Column) else _invoke_function("lit", col)
+ >>> df.select(
+ ... lit({"height": 5}).alias("data"),
+ ... lit(["python", "scala"]).alias("languages")
+ ... ).take(1)
+ [Row(data={'height': 5}, languages=['python', 'scala'])]
+ """
+ if isinstance(col, Column):
+ return col
+
+ elif isinstance(col, list):
+ return array(*[lit(x) for x in col])
+
+ elif isinstance(col, tuple):
+ fields = (
+ # Named tuple
+ col._fields if hasattr(col, "_fields")
+ # PySpark Row
+ else col.__fields__ if hasattr(col, "__fields__")
+ # Other
+ else [f"_{i + 1}" for i in range(len(col))]
+ )
+
+ return struct(*[
Review comment:
Hm, okay but these will be literals after constant folding though. @cloud-fan, WDYT about supporting complex type at literals in this way? I think we can do similar thing in Scala side as well to allow map and struct literals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769164590
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-769878358
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39244/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31207:
URL: https://github.com/apache/spark/pull/31207#issuecomment-761592216
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38725/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31207: [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31207:
URL: https://github.com/apache/spark/pull/31207#discussion_r567520322
##########
File path: python/pyspark/sql/functions.py
##########
@@ -91,13 +92,58 @@ def lit(col):
Creates a :class:`Column` of literal value.
.. versionadded:: 1.3.0
+ .. versionchanged:: 3.2.0
+ Added support for complex type literals.
+
+ Parameters
+ ----------
+ col : bool, float, int, str, datetime.date, datetime.datetime, dict, list, tuple
+ Object to be converted into :class:`Column`.
Review comment:
Ah, nvm. I missed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org