You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "devin-petersohn (via GitHub)" <gi...@apache.org> on 2026/02/18 19:35:17 UTC
[PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
devin-petersohn opened a new pull request, #54370:
URL: https://github.com/apache/spark/pull/54370
I generated some benchmarks for new implementation and compared against the old implementation. The performance numbers are show below.
- **Row counts:** 1,000 and 10,000
- **Column counts:** 2, 5, 10, 20, 40, 100
- **Data distribution:** Random uniform distribution over 10 distinct values per column
- **Total tests:** 11 configurations
| Rows | Columns | Old Time | New Time | Speedup | Time Saved | Improvement | Jobs (Old→New) | Jobs Saved |
|---------|---------|----------|----------|----------|------------|-------------|----------------|------------|
| 1,000 | **1** | **0.125s** | **0.188s** | **0.66x** | **-0.063s** | **-50.6%** | **2 → 3** | **-1** |
| 1,000 | 2 | 0.226s | 0.233s | 0.97x | -0.007s | -2.9% | 4 → 3 | 1 |
| 1,000 | 5 | 0.501s | 0.225s | 2.23x | 0.276s | 55.1% | 10 → 3 | 7 |
| 1,000 | 10 | 0.861s | 0.351s | 2.46x | 0.511s | 59.3% | 20 → 3 | 17 |
| 1,000 | 20 | 1.539s | 0.418s | 3.68x | 1.120s | 72.8% | 40 → 3 | 37 |
| 1,000 | 40 | 3.176s | 0.514s | 6.18x | 2.662s | 83.8% | 80 → 3 | 77 |
| 1,000 | 100 | 7.483s | 0.586s | 12.77x | 6.897s | 92.2% | 200 → 3 | 197 |
| 10,000 | **1** | **0.073s** | **0.111s** | **0.66x** | **-0.038s** | **-51.9%** | **2 → 3** | **-1** |
| 10,000 | 5 | 0.362s | 0.194s | 1.87x | 0.168s | 46.5% | 10 → 3 | 7 |
| 10,000 | 10 | 1.446s | 0.257s | 5.61x | 1.188s | 82.2% | 20 → 3 | 17 |
| 10,000 | 20 | 1.424s | 0.382s | 3.72x | 1.041s | 73.1% | 40 → 3 | 37 |
| 10,000 | 40 | 3.171s | 0.521s | 6.09x | 2.650s | 83.6% | 80 → 3 | 77 |
| 10,000 | 100 | 10.953s | 1.163s | 9.41x | 9.789s | 89.4% | 200 → 3 | 197 |
**Aggregate Statistics:**
- Average speedup: 4.33x
- Average improvement: 48.7%
- Average jobs saved: 54.2 per operation
- Maximum speedup: 12.77x (100 columns)
- **Regression case: 0.66x for N=1** (new approach is 50% slower)
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Fixes describe for string-only dataframes to have a fixed number of jobs rather than one job per column
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Performance
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
CI
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Co-authored-by: Claude Sonnet 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3953922431
>Is it worth it to have a fast path for single column? Just fallback the original implementation?
@gaogaotiantian I personally don't think it's worth the additional code branch, but if others have strong opinions I'm open to it. In absolute terms it is in the 10's to 100's of milliseconds of overhead, and that is just the job scheduling overhead which won't change as data scales.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3924234188
The improvement for multiple columns is great. A few questions:
1. Is it worth it to have a fast path for single column? Just fallback the original implementation?
2. If I understand correctly, we stack data right? What does it mean for memory usage when the df is large?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2833666461
##########
python/pyspark/pandas/frame.py:
##########
@@ -9899,33 +9848,39 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
+ stack_expr = f"stack({n_cols}, {stack_args}) as (column_name, value)"
+ # Unpivot, group by (column_name, value), and count occurrences
+ unpivoted = sdf.selectExpr(stack_expr)
+ value_counts = unpivoted.groupBy("column_name", "value").count()
+ # Use window function to rank values by count within each column
+ # When counts tie, pick the first value alphabetically like pandas
+ window = Window.partitionBy("column_name").orderBy(F.desc("count"), F.asc("value"))
+ # Unfortunately, there's no straightforward way to get the top value and its frequency
+ # for each column without collecting the data to the driver side.
+ top_values = (
+ value_counts.withColumn("rank", F.row_number().over(window))
+ .filter(F.col("rank") == 1)
+ .select("column_name", "value", F.col("count").alias("freq"))
+ .collect()
+ )
+ top_freq_dict = {row.column_name: (row.value, row.freq) for row in top_values}
+ tops = [str(top_freq_dict[col_name][0]) for col_name in column_names]
+ freqs = [str(top_freq_dict[col_name][1]) for col_name in column_names]
Review Comment:
Sure I can do this 🙂.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3982401066
thanks all, merged to mater
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #54370: [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1)
URL: https://github.com/apache/spark/pull/54370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3962032561
I don't have a super strong opinion about it, but 10~100 ms is probably already recognizable by human. I think the question here is how often the user will use this function with just a single column. If it's a super rare case, we don't need to care about that. If it's quite common, a fast path might be worthwhile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2824794619
##########
python/pyspark/pandas/frame.py:
##########
Review Comment:
Reason for dropping the coment?
##########
python/pyspark/pandas/frame.py:
##########
@@ -9899,33 +9848,39 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
+ stack_expr = f"stack({n_cols}, {stack_args}) as (column_name, value)"
+ # Unpivot, group by (column_name, value), and count occurrences
+ unpivoted = sdf.selectExpr(stack_expr)
+ value_counts = unpivoted.groupBy("column_name", "value").count()
+ # Use window function to rank values by count within each column
+ # When counts tie, pick the first value alphabetically like pandas
+ window = Window.partitionBy("column_name").orderBy(F.desc("count"), F.asc("value"))
+ # Unfortunately, there's no straightforward way to get the top value and its frequency
+ # for each column without collecting the data to the driver side.
Review Comment:
Note for the future: This seems like a good follow up issue, I think we could do something smarter here long term. I've been thinking about some kind of bounded collection types for aggregations and this might fit. (although tbf describe isn't used all that often, but would love to put these together if we can). They do still end up being large but on the executors and the final driver part is a bit smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3924107973
cc @gaogaotiantian FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2848487622
##########
python/pyspark/pandas/frame.py:
##########
@@ -9966,33 +9966,38 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
Review Comment:
Good catch — they are equivalent, but I agree it was confusing. Switched to using posexplode on exprs_string directly so the unpivot no longer depends on column_names similar to the other implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #54370:
URL: https://github.com/apache/spark/pull/54370#issuecomment-3946934856
> 2. If I understand correctly, we stack data right? What does it mean for memory usage when the df is large?
So eventually we'll have to collect the summaries back to the driver but the summaries will all have to fit anyways even when were executing in a loop since we store them in Python and display them. If someone had a silly number of columns this could maybe be an issue, but the old approach wouldn't work well either.
There might be a bit of extra data during the final merge steps when we're merging the aggregate objects but if that ever became an issue we could look at treeReduce (but again this would likely only happen in a degenerate case where the current implementation would also not behave well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2844213032
##########
python/pyspark/pandas/frame.py:
##########
@@ -9966,33 +9966,38 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
Review Comment:
existing implementation is using `exprs_string: List[Column]`, is it equivalent to `column_names` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2848487622
##########
python/pyspark/pandas/frame.py:
##########
@@ -9966,33 +9966,38 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
Review Comment:
Good catch — they are equivalent, but I agree it was confusing. Switched to using posexplode on exprs_string directly so the unpivot no longer depends on column_names similar to the other implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2848506888
##########
python/pyspark/pandas/frame.py:
##########
@@ -9966,33 +9966,38 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
Review Comment:
Good catch. They are equivalent, but I agree it was confusing. Switched to using posexplode on exprs_string directly so the unpivot no longer depends on column_names similar to the other implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2825277759
##########
python/pyspark/pandas/frame.py:
##########
@@ -9899,33 +9848,39 @@ def describe(self, percentiles: Optional[List[float]] = None) -> "DataFrame":
has_numeric_type = len(psser_numeric) > 0
if is_all_string_type:
- # Handling string type columns
- # We will retrieve the `count`, `unique`, `top` and `freq`.
internal = self._internal.resolved_copy
exprs_string = [
internal.spark_column_for(psser._column_label) for psser in psser_string
]
sdf = internal.spark_frame.select(*exprs_string)
- # Get `count` & `unique` for each columns
counts, uniques = map(lambda x: x[1:], sdf.summary("count", "count_distinct").take(2))
- # Handling Empty DataFrame
if len(counts) == 0 or counts[0] == "0":
data = dict()
for psser in psser_string:
data[psser.name] = [0, 0, np.nan, np.nan]
return DataFrame(data, index=["count", "unique", "top", "freq"])
- # Get `top` & `freq` for each columns
- tops = []
- freqs = []
- # TODO(SPARK-37711): We should do it in single pass since invoking Spark job
- # for every columns is too expensive.
- for column in exprs_string:
- top, freq = sdf.groupby(column).count().sort("count", ascending=False).first()
- tops.append(str(top))
- freqs.append(str(freq))
-
+ n_cols = len(column_names)
+ stack_args = ", ".join([f"'{col_name}', `{col_name}`" for col_name in column_names])
+ stack_expr = f"stack({n_cols}, {stack_args}) as (column_name, value)"
+ # Unpivot, group by (column_name, value), and count occurrences
+ unpivoted = sdf.selectExpr(stack_expr)
+ value_counts = unpivoted.groupBy("column_name", "value").count()
+ # Use window function to rank values by count within each column
+ # When counts tie, pick the first value alphabetically like pandas
+ window = Window.partitionBy("column_name").orderBy(F.desc("count"), F.asc("value"))
+ # Unfortunately, there's no straightforward way to get the top value and its frequency
+ # for each column without collecting the data to the driver side.
+ top_values = (
+ value_counts.withColumn("rank", F.row_number().over(window))
+ .filter(F.col("rank") == 1)
+ .select("column_name", "value", F.col("count").alias("freq"))
+ .collect()
+ )
+ top_freq_dict = {row.column_name: (row.value, row.freq) for row in top_values}
+ tops = [str(top_freq_dict[col_name][0]) for col_name in column_names]
+ freqs = [str(top_freq_dict[col_name][1]) for col_name in column_names]
Review Comment:
```python
top_freq_dict = {row.column_name: (str(row.value), str(row.freq)) for row in top_values}
tops, freqs = map(list, zip(*(top_freq_dict[col_name] for col_name in column_names)))
```
Maybe less duplicate? Not a huge thing though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-37711][PS] Reduce pandas describe job count from O(N) to O(1) [spark]
Posted by "devin-petersohn (via GitHub)" <gi...@apache.org>.
devin-petersohn commented on code in PR #54370:
URL: https://github.com/apache/spark/pull/54370#discussion_r2833629186
##########
python/pyspark/pandas/frame.py:
##########
Review Comment:
I can add the comments back. I removed them because I felt the comments didn't add extra context over the code (code is straightforward IMO).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org