You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "itholic (via GitHub)" <gi...@apache.org> on 2023/10/06 02:10:20 UTC
[PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
itholic opened a new pull request, #43237:
URL: https://github.com/apache/spark/pull/43237
### What changes were proposed in this pull request?
This PR proposes to fix `ps.sql` with Spark Connect.
### Why are the changes needed?
Improve the API coverage.
### Does this PR introduce _any_ user-facing change?
`ps.sql` with Pandas on Spark object now support for Spark Connect:
**Before**
```python
>>> psdf = ps.DataFrame(
... {"A": [1, 2, 3], "B": [4, 5, 6]}, index=pd.Index(["a", "b", "c"], name="index")
... )
>>> ps.sql("select * from {psdf} where A > 1", psdf=psdf)
org.apache.spark.sql.catalyst.ExtendedAnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `_pandas_api_d2d7164cc1894c8895bc7f0c541db96b` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 14;
```
**After**
```python
>>> psdf = ps.DataFrame(
... {"A": [1, 2, 3], "B": [4, 5, 6]}, index=pd.Index(["a", "b", "c"], name="index")
... )
>>> ps.sql("select * from {psdf} where A > 1", psdf=psdf)
A B
0 2 5
1 3 6
```
### How was this patch tested?
Enabling the existing test
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348172101
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
cc @zhengruifeng
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
We should still clean them no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348255353
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
It is a known issue, we cann't drop the temp view based on existing framework, and it will pollute the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS][DOCS] Add note for `ps.sql` [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #43237:
URL: https://github.com/apache/spark/pull/43237#issuecomment-1754486647
Applied suggestions & updated PR title/description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348301213
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
Thanks for checking!
Then I think maybe we can choose one of:
1. Support with proper warning that the catalog could be polluted.
e.g.
```
Temp view `_pandas_api_3ee629ad38024b64bb9301b24315fd36` is created when performing `ps.sql` and it could pollute the catalog list. Please manually remove the temp view by running `spark.catalog.dropTempView('_pandas_api_3ee629ad38024b64bb9301b24315fd36')` when the resulting DataFrame no longer be used.
```
2. Not support with proper note why we currently don't support.
e.g.
```
`ps.sql` currently does not work with pandas-on-Spark object on Spark Connect because it creates the random temp view that could pollute the catalog list.
```
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348233279
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
> it shows a bunch of random tables.
Ah, I see. Then let's wait for another look from @zhengruifeng 👀
Or maybe we could just prevent using pandas-on-Spark object for `ps.sql` with Spark Connect for now, and document why we currently doesn't support this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1351855386
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -201,8 +203,7 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- if not is_remote():
Review Comment:
Sounds good. Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348301213
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
Thanks for checking!
Then I think maybe we can choose one of:
1. Support with proper warning that the catalog may be polluted.
2. Not support with proper note why we currently don't support.
WDYT?
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
Thanks for checking!
Then I think maybe we can choose one of:
1. Support with proper warning that the catalog could be polluted.
2. Not support with proper note why we currently don't support.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348301213
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
Thanks for checking!
Then I think maybe we can choose one of:
1. Support with proper warning that the catalog may be polluted due to a random temp view.
2. Not support with proper note why we currently don't support.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #43237:
URL: https://github.com/apache/spark/pull/43237#issuecomment-1759147891
CI passed. @HyukjinKwon @zhengruifeng FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43237: [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect.
URL: https://github.com/apache/spark/pull/43237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43237:
URL: https://github.com/apache/spark/pull/43237#issuecomment-1759385925
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348233279
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
> it shows a bunch of random tables.
Ah, I see. Then let's wait for another look from @zhengruifeng 👀
Or maybe we could just prevent using pandas-on-Spark object for `ps.sql` with Spark Connect for now, and document why we currently doesn't support this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1351868056
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -59,6 +61,9 @@ def sql(
Also the method can bind named parameters to SQL literals from `args`.
+ .. note::
+ pandas-on-Spark objects are not supported for Spark Connect currently.
Review Comment:
```suggestion
pandas-on-Spark objects are not supported for Spark Connect.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348226460
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
IMHO it is okay for now since the possibility of side effects is significantly low enough to ignore, since the name of temp view is generated by using `uuid.uuid4()` so it's almost no chance of overlap, and also all the temp views are automatically deleted when the session ends.
Yeah, but I still agree that ideally it would be best to ensure that the temp view operates properly even after it is deleted from the remote session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348172327
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
We should keep the temp view for Spark Connect after formatting. Otherwise, it fails to find a view properly.
```
org.apache.spark.sql.catalyst.ExtendedAnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `_pandas_api_d2d7164cc1894c8895bc7f0c541db96b` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 14;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348233279
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
> it shows a bunch of random tables.
Ah, I see. Then let's wait for another look from @zhengruifeng 👀
And maybe we could just prevent using pandas-on-Spark object for `ps.sql` with Spark Connect for now, and document why we currently doesn't support this.
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
> it shows a bunch of random tables.
Ah, I see. Then let's wait for another look from @zhengruifeng 👀
And maybe we could just prevent using pandas-on-Spark object for `ps.sql` with Spark Connect for now, and document why we currently doesn't support this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Fix `ps.sql` for remote session [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1348228957
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -200,7 +201,8 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- formatter.clear()
+ if not is_remote():
+ formatter.clear()
Review Comment:
Nah, it will pollute all the catalog. Think about the case like `SHOW TABLES`, and it shows a bunch of random tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS][DOCS] Add note for `ps.sql` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #43237:
URL: https://github.com/apache/spark/pull/43237#discussion_r1351696061
##########
python/pyspark/pandas/sql_formatter.py:
##########
@@ -201,8 +203,7 @@ def sql(
try:
sdf = session.sql(formatter.format(query, **kwargs), args)
finally:
- if not is_remote():
Review Comment:
Can we throw an exception instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect. [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #43237:
URL: https://github.com/apache/spark/pull/43237#issuecomment-1756625801
Oh, seems like only `ps.DataFrame` is not working properly (ps.Series works fine). Just updated related places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org