You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/15 00:15:15 UTC
[GitHub] [spark] xinrong-meng opened a new pull request, #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
xinrong-meng opened a new pull request, #37888:
URL: https://github.com/apache/spark/pull/37888
### What changes were proposed in this pull request?
Consolidate `lit` function with NumPy scalar in sql and pandas module.
Major changes:
- More accurate types for `lit` in sql with NumPy scalar input
- Replace `lit` in pandas with `lit` in sql
### Why are the changes needed?
As part of [SPARK-39405](https://issues.apache.org/jira/browse/SPARK-39405) for NumPy support in SQL.
That provides more precise and consistent Numpy support across PySpark moduels.
### Does this PR introduce _any_ user-facing change?
Yes. More accurate types for `lit`.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1248987419
```
test_repeat (pyspark.pandas.tests.test_spark_functions.SparkFunctionsTests) ... FAIL (0.052s)
======================================================================
FAIL [0.052s]: test_repeat (pyspark.pandas.tests.test_spark_functions.SparkFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/pandas/tests/test_spark_functions.py", line 28, in test_repeat
self.assertTrue(spark_column_equals(SF.repeat(F.lit(1), 2), F.repeat(F.lit(1), 2)))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 8.471s
```
Still failed due to `test_repeat`, Initial invistigation:
```
F.repeat(F.lit(1), 2).__dict__
Out[3]: {'_jc': JavaObject id=o50}
SF.repeat(F.lit(1), 2).__dict__
Out[4]: {'_jc': JavaObject id=o54}
```
According to https://github.com/apache/spark/pull/37888#discussion_r971408190 , Looks like we need to remove this test?
```python
class SparkFunctionsTests(PandasOnSparkTestCase):
def test_repeat(self):
# TODO: Placeholder
pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1248886104
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1247678473
Btw, I think we can further support `lit(np.ndarray)` like [`lit([...])`](https://github.com/apache/spark/pull/37722) in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1247704982
Can we do this ^ @itholic ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1247433538
CC @ueshin @zhengruifeng @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1248909625
this PR doesn't cover the newly added `groupby.quantile`:
```
starting python compilation test...
python compilation succeeded.
starting black test...
black checks passed.
starting flake8 test...
flake8 checks passed.
starting mypy annotations test...
annotations failed mypy checks:
python/pyspark/pandas/window.py:112: error: Module has no attribute "lit" [attr-defined]
Found 1 error in 1 file (checked 340 source files)
1
```
fixed in https://github.com/apache/spark/pull/37908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
URL: https://github.com/apache/spark/pull/37888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
URL: https://github.com/apache/spark/pull/37888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on code in PR #37888:
URL: https://github.com/apache/spark/pull/37888#discussion_r971408190
##########
python/pyspark/pandas/tests/test_spark_functions.py:
##########
@@ -20,25 +20,12 @@
from pyspark.pandas.spark import functions as SF
from pyspark.pandas.utils import spark_column_equals
from pyspark.sql import functions as F
-from pyspark.sql.types import (
- ByteType,
- FloatType,
- IntegerType,
- LongType,
-)
from pyspark.testing.pandasutils import PandasOnSparkTestCase
class SparkFunctionsTests(PandasOnSparkTestCase):
- def test_lit(self):
- self.assertTrue(spark_column_equals(SF.lit(np.int64(1)), F.lit(1).astype(LongType())))
- self.assertTrue(spark_column_equals(SF.lit(np.int32(1)), F.lit(1).astype(IntegerType())))
- self.assertTrue(spark_column_equals(SF.lit(np.int8(1)), F.lit(1).astype(ByteType())))
- self.assertTrue(spark_column_equals(SF.lit(np.byte(1)), F.lit(1).astype(ByteType())))
- self.assertTrue(
- spark_column_equals(SF.lit(np.float32(1)), F.lit(float(1)).astype(FloatType()))
- )
- self.assertTrue(spark_column_equals(SF.lit(1), F.lit(1)))
+ def test_repeat(self):
Review Comment:
This change is irrelevant to the PR proposal but is made to keep the `test_spark_functions` file.
We shall complete tests for `pyspark/pandas/spark/functions` here later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1251487641
Thank you @HyukjinKwon @zhengruifeng @Yikun for taking care of the merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1247424459
It may be easier to review by commits :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1248800184
Sure, just created ticket: https://issues.apache.org/jira/browse/SPARK-40462
Let me take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37888: [SPARK-40196][PYTHON][PS] Consolidate `lit` function with NumPy scalar in sql and pandas module
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37888:
URL: https://github.com/apache/spark/pull/37888#issuecomment-1248990425
@Yikun Let's comment this test for now, to unblock other PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org