You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2024/02/15 23:20:02 UTC
[PR] [WIP] Consolidated API for V2 profiling [spark]
xinrong-meng opened a new pull request, #45129:
URL: https://github.com/apache/spark/pull/45129
### What changes were proposed in this pull request?
spark.profile.show/dump
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes. spark.profile.show/dump are supported.
### How was this patch tested?
Unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1491866174
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
+ """
+ Show the profile results.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ type : str, optional
+ The profiler type, which can be either "perf" or "memory".
+ id : int, optional
+ A UDF ID to be shown. If not specified, all the results will be shown.
+ """
+ if type == "memory":
+ self.sparkSession.showMemoryProfiles(id)
Review Comment:
Shall we remove the old APIs? I think the new APIs are enough to have.
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
Review Comment:
It should take `ProfilerCollector`?
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
+ """
+ Show the profile results.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ type : str, optional
+ The profiler type, which can be either "perf" or "memory".
+ id : int, optional
+ A UDF ID to be shown. If not specified, all the results will be shown.
+ """
+ if type == "memory":
+ self.sparkSession.showMemoryProfiles(id)
+ elif type == "perf" or type is None:
+ self.sparkSession.showPerfProfiles(id)
+ if type is None: # Show both perf and memory profiles
+ self.sparkSession.showMemoryProfiles(id)
+ else:
+ raise PySparkValueError(
+ error_class="VALUE_NOT_ALLOWED",
+ message_parameters={
+ "arg_name": "type",
+ "allowed_values": str(["perf", "memory"]),
+ },
+ )
+
+ def dump(self, path: str, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
Review Comment:
ditto.
```suggestion
def dump(self, path: str, id: Optional[int] = None, *, type: Optional[str] = None) -> None:
```
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
Review Comment:
I prefer:
```suggestion
def show(self, id: Optional[int] = None, *, type: Optional[str] = None) -> None:
```
```py
spark.profile.show() # show all profile results
spark.profile.show(1) # show the profile results for ID = 1
spark.profile.show(1, type="memory") # show the memory profile results for ID = 1
spark.profile.show(type="memory") # show all memory profile results
```
##########
python/pyspark/sql/session.py:
##########
@@ -906,6 +907,12 @@ def dataSource(self) -> "DataSourceRegistration":
return DataSourceRegistration(self)
+ @property
+ def profile(self) -> "Profile":
Review Comment:
Need this for connect, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1496339880
##########
python/pyspark/sql/session.py:
##########
@@ -76,6 +76,7 @@
from pyspark.sql.udf import UDFRegistration
from pyspark.sql.udtf import UDTFRegistration
from pyspark.sql.datasource import DataSourceRegistration
+ from pyspark.sql.profiler import Profile
Review Comment:
Adjusted. I noticed we intended to import inside a `property` as well as under TYPE_CHECKING, like `dataSource` and `udtf`, `udf`, etc. But now I believe that is not a convention.
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +240,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, profiler_collector: "ProfilerCollector"):
Review Comment:
Adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #45129:
URL: https://github.com/apache/spark/pull/45129#issuecomment-1960153150
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #45129:
URL: https://github.com/apache/spark/pull/45129#issuecomment-1960617435
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1496336908
##########
python/pyspark/sql/connect/profiler.py:
##########
@@ -39,3 +39,14 @@ def _profile_results(self) -> "ProfileResults":
def _update(self, update: "ProfileResults") -> None:
with self._lock:
self._value = ProfileResultsParam.addInPlace(self._profile_results, update)
+
+
+class ConnectProfile(Profile):
Review Comment:
Good catch! Adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #45129:
URL: https://github.com/apache/spark/pull/45129#issuecomment-1954865462
Had to rebase the master change, the latest changes after review are [ad98eb3](https://github.com/apache/spark/pull/45129/commits/ad98eb39aa4da75258365a42d74c093692dcb9f0) and [46118f9](https://github.com/apache/spark/pull/45129/commits/46118f900507b8b48d034dca6cae7b382b04658f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1496340148
##########
python/pyspark/sql/session.py:
##########
@@ -906,6 +907,12 @@ def dataSource(self) -> "DataSourceRegistration":
return DataSourceRegistration(self)
+ @property
+ def profile(self) -> "Profile":
Review Comment:
Adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1493048620
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
Review Comment:
Yes after removing the old APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1491924933
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
Review Comment:
We should probably also fix the docs at `python/docs/source/reference`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #45129: [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling
URL: https://github.com/apache/spark/pull/45129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1491925413
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
+ """
+ Show the profile results.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ type : str, optional
+ The profiler type, which can be either "perf" or "memory".
+ id : int, optional
+ A UDF ID to be shown. If not specified, all the results will be shown.
+ """
+ if type == "memory":
+ self.sparkSession.showMemoryProfiles(id)
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1493050589
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
Review Comment:
Good point! Would you mind if I file a follow-up pr for documentation? We might also want to edit the PySpark debugging guide together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1493048255
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
Review Comment:
Your suggestion makes more sense. Thanks!
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
+ """
+ Show the profile results.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ type : str, optional
+ The profiler type, which can be either "perf" or "memory".
+ id : int, optional
+ A UDF ID to be shown. If not specified, all the results will be shown.
+ """
+ if type == "memory":
+ self.sparkSession.showMemoryProfiles(id)
+ elif type == "perf" or type is None:
+ self.sparkSession.showPerfProfiles(id)
+ if type is None: # Show both perf and memory profiles
+ self.sparkSession.showMemoryProfiles(id)
+ else:
+ raise PySparkValueError(
+ error_class="VALUE_NOT_ALLOWED",
+ message_parameters={
+ "arg_name": "type",
+ "allowed_values": str(["perf", "memory"]),
+ },
+ )
+
+ def dump(self, path: str, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
Review Comment:
Adjusted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1493048037
##########
python/pyspark/sql/profiler.py:
##########
@@ -239,3 +241,72 @@ def _profile_results(self) -> "ProfileResults":
with self._lock:
value = self._accumulator.value
return value if value is not None else {}
+
+
+class Profile:
+ """User-facing profile API. This instance can be accessed by
+ :attr:`spark.profile`.
+
+ .. versionadded: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def show(self, *, type: Optional[str] = None, id: Optional[int] = None) -> None:
+ """
+ Show the profile results.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ type : str, optional
+ The profiler type, which can be either "perf" or "memory".
+ id : int, optional
+ A UDF ID to be shown. If not specified, all the results will be shown.
+ """
+ if type == "memory":
+ self.sparkSession.showMemoryProfiles(id)
Review Comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47069][PYTHON] Introduce `spark.profile.show/dump` for SparkSession-based profiling [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #45129:
URL: https://github.com/apache/spark/pull/45129#discussion_r1493049117
##########
python/pyspark/sql/session.py:
##########
@@ -906,6 +907,12 @@ def dataSource(self) -> "DataSourceRegistration":
return DataSourceRegistration(self)
+ @property
+ def profile(self) -> "Profile":
Review Comment:
I planned to make a follow-up PR for Connect. But now I think I'll just add Connect changes to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org