You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/07/25 03:37:26 UTC

[GitHub] [spark] zhengruifeng opened a new pull request, #42146: [WIP][DO_NOT_MERGE][INFRA] Disable python packaging tests

zhengruifeng opened a new pull request, #42146:
URL: https://github.com/apache/spark/pull/42146

   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1273233607


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   and I see packaging test in `pyspark-core` successed in https://github.com/apache/spark/actions/runs/5652170312/job/15311427056, so maybe packaging test itself is fine?
   
   TBH, I don't figure out what is happening



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #42146: [SPARK-44544][INFRA] Move python packaging tests to a separate module

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1650778187

   let me take a quick look at https://github.com/apache/spark/pull/42159 too. I think we can just remove some directory before running pip test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1651049025

   @HyukjinKwon I think this PR is orthogonal to https://github.com/apache/spark/pull/42159:
   this PR now aims to only trigger `run_python_packaging_tests` in one pyspark test module, it seem ok to resolve the CI failure;
   https://github.com/apache/spark/pull/42159 is to reduce the disk requirement of `run_python_packaging_tests`, which can make the CI more stable, IIUC
   
   shall we go with this PR first (after all tests pass)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #42146: [SPARK-44544][INFRA] Move python packaging tests to a separate module

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1650857362

   yeah. let me take a quick look at https://github.com/apache/spark/pull/42159


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1651673288

   @panbingkun good catch! Let me backport it to 3.4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1273231043


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   I am not 100% sure about this, since in https://github.com/zhengruifeng/spark/actions/runs/5652363908/job/15311889590 I also disabled all the packaging tests. (you can check that there is no packaging tests in `pyspark-core`),
   but `pyspark-sql` and `pyspark-pandas-slow-connect` still failed ...
   
   
   I just rebase the PR to re-test whether disabling works



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on pull request #42146: [SPARK-44544][INFRA] Move python packaging tests to a separate module

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1650810404

   @zhengruifeng @HyukjinKwon 
   Perhaps we can disable 'run_python_packaging_tests' first to allow GA to quickly recover, and then handle this matter later on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1649754172

   It seems disabling `run_python_packaging_tests` works, let me add it back in a separate job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1273225774


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   <img width="1026" alt="image" src="https://github.com/apache/spark/assets/15246973/7f4b4b74-26d8-4bb9-aec9-b65367c3de96">
   
   From this workflow: https://github.com/panbingkun/spark/actions/runs/5653268655/job/15314215187 Look, it seems to be a problem with `run_python_packaging_tests`.
   
   Because all the tests in `pyspark-pandas-connect-part-3` have ended running, `run_python_packaging_tests` caused an error



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1273403626


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   From the test results, after disable `run_python_packaging_tests` and splitting, it was successful.
   https://github.com/panbingkun/spark/actions/runs/5654907063/job/15318981268



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Disable python packaging tests

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1272968497


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   I'm just checking whether disabling this test works.
   will do so if this is the root cause.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng closed pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`
URL: https://github.com/apache/spark/pull/42146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1651139374

   also cc @LuciferYang @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42146: [WIP][DO_NOT_MERGE][INFRA] Disable python packaging tests

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42146:
URL: https://github.com/apache/spark/pull/42146#discussion_r1272965815


##########
dev/run-tests.py:
##########
@@ -655,7 +655,7 @@ def main():
             opts.parallelism,
             with_coverage=os.environ.get("PYSPARK_CODECOV", "false") == "true",
         )
-        run_python_packaging_tests()
+        # run_python_packaging_tests()

Review Comment:
   Yeah, we should probably deduplicate because it runs for every python module test. another way is just to add an env variable, and enable it in only one split.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [SPARK-44544][INFRA] Move python packaging tests to a separate module

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1649814696

   also cc @Yikun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1651170091

   all python tests passed, merged to master/branch-3.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on pull request #42146: [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #42146:
URL: https://github.com/apache/spark/pull/42146#issuecomment-1651222837

   > all python tests passed, merged to master/branch-3.5
   
   @zhengruifeng Maybe branch-3.4 also need it?
   https://github.com/panbingkun/spark/actions/runs/5665345967/job/15350140605


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org