You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2025/11/11 19:30:12 UTC
[PR] [SPARK-54297] Optimize tests for iterator of dataframe API in `applyInPandas` [spark]
Yicong-Huang opened a new pull request, #53002:
URL: https://github.com/apache/spark/pull/53002
### What changes were proposed in this pull request?
We use these tests to ensure that large datasets are processed exclusively through the new iterator API and that loading them entirely into memory would fail. However, the original tests were too slow and resource-intensive. This PR optimizes the test to achieve the same validation goal while using significantly fewer resources.
### Why are the changes needed?
To save time
Before:
```
7d04fd12050e/python3.11__pyspark.sql.tests.connect.pandas.test_parity_pandas_grouped_map__6iwxrtx4.log)
Finished test(python3.11): pyspark.sql.tests.connect.pandas.test_parity_pandas_grouped_map **(291s)**
Starting test(python3.11): pyspark.sql.tests.pandas.test_pandas_grouped_map (temp output: /__w/spark/spark/python/target/379cea6f-7421-4f81-ae7b-1c22baddaa92/python3.11__pyspark.sql.tests.pandas.test_pandas_grouped_map__km0800fd.log)
Finished test(python3.11): pyspark.sql.tests.pandas.test_pandas_grouped_map **(300s)**
```
After:
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI test.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54297][PYTHON][SQL][TESTS] Optimize tests for iterator of dataframe API in `applyInPandas` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #53002: [SPARK-54297][PYTHON][SQL][TESTS] Optimize tests for iterator of dataframe API in `applyInPandas`
URL: https://github.com/apache/spark/pull/53002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54297][PYTHON][SQL][TESTS] Optimize tests for iterator of dataframe API in `applyInPandas` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #53002:
URL: https://github.com/apache/spark/pull/53002#issuecomment-3520223608
late LGTM, thanks @Yicong-Huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org