You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/21 08:33:38 UTC

[GitHub] [spark] Yikun opened a new pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Yikun opened a new pull request #35595:
URL: https://github.com/apache/spark/pull/35595


   ### What changes were proposed in this pull request?
   Change `docker-for-desktop` to `docker-desktop`.
   
   
   ### Why are the changes needed?
   The context name of the kubernetes on docker for desktop should be `docker-desktop` rather than `docker-for-desktop`
   
   ```
   $ k config current-context
   docker-desktop
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   - CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047525818


   @Yikun . We have old Docker Desktop users and installations. Not everyone updates their SWs on the zero days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047535468


   @dongjoon-hyun Yes, agree, that's also why I revert my patch as your suggestion. : )
   
   To clarify and for followup developer reference, just note here: According to the [comments](https://github.com/docker/for-win/issues/5089#issuecomment-582752325),  since docker desktop v2.4 (current is v4.5.1), `docker` are using use a alias `docker-for-desktop` to link `docker-desktop` cluster for legacy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1046643923


   I also take a look why "docker-for-backend" is used before, looks like there are [some renaming work ](https://github.com/docker/for-mac/issues/4089) on `docker-for-backend`. After this, we can upgrade kubernetes v5.12.1 safely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1048400674


   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r812221664



##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -172,7 +172,7 @@ to the wrapper scripts and using the wrapper scripts will simply set these appro
     <td><code>spark.kubernetes.test.deployMode</code></td>
     <td>
       The integration test backend to use.  Acceptable values are <code>minikube</code>, 
-      <code>docker-for-desktop</code> and <code>cloud</code>.
+      <code>docker-desktop</code> and <code>cloud</code>.

Review comment:
       Looks good.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1046643923


   I also take a look why "docker-for-backend" is used before, looks like there are [some renaming work ](https://github.com/docker/for-mac/issues/4089) on `docker-for-backend`, so I change all `docker-for-desktop` to `docker-desktop` in this patch.
   
   After this patch, we can upgrade kubernetes v5.12.1 safely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1046625236


   cc @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r811429497



##########
File path: resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
##########
@@ -136,7 +136,7 @@ then
       fi
       ;;
 
-    docker-for-desktop)
+    docker-desktop)

Review comment:
       I'd have both to avoid any surprise on the previous test pipeline.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r812221802



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala
##########
@@ -43,7 +43,7 @@ private[spark] object IntegrationTestBackendFactory {
       case BACKEND_MINIKUBE => MinikubeTestBackend
       case BACKEND_CLOUD =>
         new KubeConfigBackend(System.getProperty(CONFIG_KEY_KUBE_CONFIG_CONTEXT))
-      case BACKEND_DOCKER_FOR_DESKTOP => DockerForDesktopBackend
+      case BACKEND_DOCKER_FOR_DESKTOP | BACKEND_DOCKER_DESKTOP => DockerForDesktopBackend

Review comment:
       Yes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r811429652



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
##########
@@ -71,7 +71,7 @@ private[spark] trait PVTestsSuite { k8sSuite: KubernetesSuite =>
                 .withMatchExpressions(new NodeSelectorRequirementBuilder()
                   .withKey("kubernetes.io/hostname")
                   .withOperator("In")
-                  .withValues("minikube", "m01", "docker-for-desktop", "docker-desktop")
+                  .withValues("minikube", "m01", "docker-desktop")

Review comment:
       Please don't remove this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047384625


   @dongjoon-hyun Sure, it make sense to keep test cmd compatible. Will adress soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] codecov-commenter commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047681956


   # [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#35595](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48b56c0) into [master](https://codecov.io/gh/apache/spark/commit/ae67adde4d2dc0a75e03710fc3e66ea253feeda3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ae67add) will **decrease** coverage by `29.59%`.
   > The diff coverage is `50.00%`.
   
   > :exclamation: Current head 48b56c0 differs from pull request most recent head 8fbcc3c. Consider uploading reports for the commit 8fbcc3c to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/35595/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #35595       +/-   ##
   ===========================================
   - Coverage   91.22%   61.63%   -29.60%     
   ===========================================
     Files         297      202       -95     
     Lines       64441    40269    -24172     
     Branches     9914     7525     -2389     
   ===========================================
   - Hits        58789    24821    -33968     
   - Misses       4288    14233     +9945     
   + Partials     1364     1215      -149     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `61.63% <50.00%> (-29.57%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/context.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvY29udGV4dC5weQ==) | `46.99% <50.00%> (-39.99%)` | :arrow_down: |
   | [python/pyspark/join.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvam9pbi5weQ==) | `12.12% <0.00%> (-81.82%)` | :arrow_down: |
   | [python/pyspark/sql/observation.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3FsL29ic2VydmF0aW9uLnB5) | `26.08% <0.00%> (-69.57%)` | :arrow_down: |
   | [python/pyspark/ml/tuning.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvdHVuaW5nLnB5) | `25.03% <0.00%> (-67.46%)` | :arrow_down: |
   | [python/pyspark/pandas/frame.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2ZyYW1lLnB5) | `33.03% <0.00%> (-64.03%)` | :arrow_down: |
   | [python/pyspark/streaming/dstream.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3RyZWFtaW5nL2RzdHJlYW0ucHk=) | `22.65% <0.00%> (-61.72%)` | :arrow_down: |
   | [python/pyspark/util.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvdXRpbC5weQ==) | `24.78% <0.00%> (-60.69%)` | :arrow_down: |
   | [python/pyspark/streaming/util.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc3RyZWFtaW5nL3V0aWwucHk=) | `29.62% <0.00%> (-60.50%)` | :arrow_down: |
   | [python/pyspark/resource/requests.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcmVzb3VyY2UvcmVxdWVzdHMucHk=) | `35.25% <0.00%> (-60.44%)` | :arrow_down: |
   | [python/pyspark/cloudpickle/compat.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvY2xvdWRwaWNrbGUvY29tcGF0LnB5) | `30.00% <0.00%> (-60.00%)` | :arrow_down: |
   | ... and [192 more](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8985427...8fbcc3c](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047535468


   @dongjoon-hyun Yes, that's also why my revert my patch as your suggestion. : )
   
   To clarify and for followup developer reference, according to the [comments](https://github.com/docker/for-win/issues/5089#issuecomment-582752325), `docker` are using use a alias `docker-for-desktop` to link `docker-desktop` cluster for legacy. Since docker v2.4 (current is v4.5.1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35595:
URL: https://github.com/apache/spark/pull/35595


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047384625


   @dongjoon-hyun Sure, it make sense to keep test cmd compatible. Will address soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode`, make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` context.
   
   And drop the docker version support before 2.4.
   
   The test cmd (docker-desktop/docker-for-desktop) passed in my env:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1046643923


   I also take a look why "docker-for-backend" is used before, looks like there are [some renaming work ](https://github.com/docker/for-mac/issues/4089) on `docker-for-backend`. After this patch, we can upgrade kubernetes v5.12.1 safely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] codecov-commenter edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047681956


   # [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#35595](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48b56c0) into [master](https://codecov.io/gh/apache/spark/commit/ae67adde4d2dc0a75e03710fc3e66ea253feeda3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ae67add) will **decrease** coverage by `17.01%`.
   > The diff coverage is `80.55%`.
   
   > :exclamation: Current head 48b56c0 differs from pull request most recent head 8fbcc3c. Consider uploading reports for the commit 8fbcc3c to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/35595/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #35595       +/-   ##
   ===========================================
   - Coverage   91.22%   74.21%   -17.02%     
   ===========================================
     Files         297      248       -49     
     Lines       64441    49136    -15305     
     Branches     9914     8516     -1398     
   ===========================================
   - Hits        58789    36464    -22325     
   - Misses       4288    11253     +6965     
   - Partials     1364     1419       +55     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `74.20% <80.55%> (-17.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/context.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvY29udGV4dC5weQ==) | `61.34% <50.00%> (-25.64%)` | :arrow_down: |
   | [python/pyspark/mllib/tree.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWxsaWIvdHJlZS5weQ==) | `88.88% <82.35%> (-3.04%)` | :arrow_down: |
   | [python/pyspark/join.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvam9pbi5weQ==) | `12.12% <0.00%> (-81.82%)` | :arrow_down: |
   | [python/pyspark/ml/tuning.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvdHVuaW5nLnB5) | `25.03% <0.00%> (-67.46%)` | :arrow_down: |
   | [python/pyspark/pandas/frame.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2ZyYW1lLnB5) | `33.03% <0.00%> (-64.03%)` | :arrow_down: |
   | [python/pyspark/ml/pipeline.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvcGlwZWxpbmUucHk=) | `35.53% <0.00%> (-59.40%)` | :arrow_down: |
   | [python/pyspark/pandas/series.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL3Nlcmllcy5weQ==) | `39.85% <0.00%> (-56.08%)` | :arrow_down: |
   | [python/pyspark/util.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvdXRpbC5weQ==) | `30.76% <0.00%> (-54.71%)` | :arrow_down: |
   | [python/pyspark/shuffle.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc2h1ZmZsZS5weQ==) | `18.73% <0.00%> (-53.68%)` | :arrow_down: |
   | [python/pyspark/pandas/generic.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcGFuZGFzL2dlbmVyaWMucHk=) | `37.02% <0.00%> (-50.96%)` | :arrow_down: |
   | ... and [114 more](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8985427...8fbcc3c](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047514625


   BTW, the `docker-for-desktop` has been already deprecated in docker official site, if really needed, we might better do some rename things in code in some future time.
   
   Anyway, let's first fix it. : )
   
   [1] https://www.docker.com/products/docker-desktop


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047514625


   BTW, the `docker-for-desktop` has been already deprecated in docker official site, if needed, we might better do some rename things in code to avoid confused in some future time.
   
   Anyway, let's first fix it. : )
   
   [1] https://www.docker.com/products/docker-desktop


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r812218422



##########
File path: resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
##########
@@ -136,7 +136,7 @@ then
       fi
       ;;
 
-    docker-desktop)
+    docker-desktop | docker-for-desktop)

Review comment:
       Yes, this one is what I asked.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047535468


   @dongjoon-hyun Yes, agree, that's also why I revert my patch as your suggestion. : )
   
   To clarify and for followup developer reference, according to the [comments](https://github.com/docker/for-win/issues/5089#issuecomment-582752325),  Since docker v2.4 (current is v4.5.1), `docker` are using use a alias `docker-for-desktop` to link `docker-desktop` cluster for legacy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047638100


   Here is the error message in case of the old deployMode name.
   ```
   [info] KubernetesSuite:
   [info] org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED *** (51 milliseconds)
   [info]   java.lang.IllegalArgumentException: Invalid spark.kubernetes.test.deployMode: docker-for-desktop
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode`, make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` backend.
   
   And drop the docker version support before 2.4.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047535468


   @dongjoon-hyun Yes, agree, that's also why I revert my patch as your suggestion. : )
   
   To clarify and for followup developer reference, according to the [comments](https://github.com/docker/for-win/issues/5089#issuecomment-582752325), `docker` are using use a alias `docker-for-desktop` to link `docker-desktop` cluster for legacy. Since docker v2.4 (current is v4.5.1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on a change in pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35595:
URL: https://github.com/apache/spark/pull/35595#discussion_r810927983



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
##########
@@ -71,7 +71,7 @@ private[spark] trait PVTestsSuite { k8sSuite: KubernetesSuite =>
                 .withMatchExpressions(new NodeSelectorRequirementBuilder()
                   .withKey("kubernetes.io/hostname")
                   .withOperator("In")
-                  .withValues("minikube", "m01", "docker-for-desktop", "docker-desktop")
+                  .withValues("minikube", "m01", "docker-desktop")

Review comment:
       Note: `docker-for-desktop` was already [renamed to](https://github.com/docker/for-mac/issues/4089) `docker-desktop`  since docker desktop 2.4 for win and mac, so we can drop `docker-for-desktop` in here safely.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047514625


   BTW, the `docker-for-desktop` has been already deprecated in docker official site, if needed, we might better do some rename things in code in some future time, to avoid confused .
   
   Anyway, let's first fix it. : )
   
   [1] https://www.docker.com/products/docker-desktop


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047514625


   BTW, the `docker-for-desktop` has been already deprecated in docker official site, if needed, we might better do some rename things in code in some future time.
   
   Anyway, let's first fix it. : )
   
   [1] https://www.docker.com/products/docker-desktop


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047638100


   Here is the error message when you use the old deployMode name.
   ```
   [info] KubernetesSuite:
   [info] org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED *** (51 milliseconds)
   [info]   java.lang.IllegalArgumentException: Invalid spark.kubernetes.test.deployMode: docker-for-desktop
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode`, make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` context.
   
   And drop the docker version support before 2.4.
   
   I tested the cmd according PR description in my env.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode`, make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` context.
   
   And drop the docker version support before 2.4.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047384625


   @dongjoon-hyun Sure, make sense to keep test cmd compatible. Will address soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun closed pull request #35595: [SPARK-38272][K8S][TESTS] Fix wrong context name for integration test on docker-desktop

Posted by GitBox <gi...@apache.org>.
Yikun closed pull request #35595:
URL: https://github.com/apache/spark/pull/35595


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] codecov-commenter edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047681956


   # [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#35595](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (48b56c0) into [master](https://codecov.io/gh/apache/spark/commit/ae67adde4d2dc0a75e03710fc3e66ea253feeda3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ae67add) will **decrease** coverage by `7.66%`.
   > The diff coverage is `80.55%`.
   
   > :exclamation: Current head 48b56c0 differs from pull request most recent head 8fbcc3c. Consider uploading reports for the commit 8fbcc3c to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/spark/pull/35595/graphs/tree.svg?width=650&height=150&src=pr&token=R9pHLWgWi8&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #35595      +/-   ##
   ==========================================
   - Coverage   91.22%   83.56%   -7.67%     
   ==========================================
     Files         297      257      -40     
     Lines       64441    58755    -5686     
     Branches     9914     9305     -609     
   ==========================================
   - Hits        58789    49099    -9690     
   - Misses       4288     8451    +4163     
   + Partials     1364     1205     -159     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `83.54% <80.55%> (-7.66%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [python/pyspark/context.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvY29udGV4dC5weQ==) | `61.34% <50.00%> (-25.64%)` | :arrow_down: |
   | [python/pyspark/mllib/tree.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWxsaWIvdHJlZS5weQ==) | `88.88% <82.35%> (-3.04%)` | :arrow_down: |
   | [python/pyspark/join.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvam9pbi5weQ==) | `12.12% <0.00%> (-81.82%)` | :arrow_down: |
   | [python/pyspark/ml/tuning.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvdHVuaW5nLnB5) | `25.03% <0.00%> (-67.46%)` | :arrow_down: |
   | [python/pyspark/ml/pipeline.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvcGlwZWxpbmUucHk=) | `35.53% <0.00%> (-59.40%)` | :arrow_down: |
   | [python/pyspark/util.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvdXRpbC5weQ==) | `30.76% <0.00%> (-54.71%)` | :arrow_down: |
   | [python/pyspark/shuffle.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3Bhcmsvc2h1ZmZsZS5weQ==) | `18.73% <0.00%> (-53.68%)` | :arrow_down: |
   | [python/pyspark/ml/image.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvaW1hZ2UucHk=) | `33.33% <0.00%> (-50.01%)` | :arrow_down: |
   | [python/pyspark/rdd.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvcmRkLnB5) | `42.34% <0.00%> (-47.94%)` | :arrow_down: |
   | [python/pyspark/ml/wrapper.py](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHl0aG9uL3B5c3BhcmsvbWwvd3JhcHBlci5weQ==) | `45.85% <0.00%> (-47.32%)` | :arrow_down: |
   | ... and [81 more](https://codecov.io/gh/apache/spark/pull/35595/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8985427...8fbcc3c](https://codecov.io/gh/apache/spark/pull/35595?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047535468


   @dongjoon-hyun Yes, that's also why I revert my patch as your suggestion. : )
   
   To clarify and for followup developer reference, according to the [comments](https://github.com/docker/for-win/issues/5089#issuecomment-582752325), `docker` are using use a alias `docker-for-desktop` to link `docker-desktop` cluster for legacy. Since docker v2.4 (current is v4.5.1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode`, make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` context.
   
   And drop the docker version support before 2.4.
   
   The test cmd (docker-desktop/docker-for-desktop) passed in my env.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1057343582


   I tested and backported this to branch-3.2 for Apache Spark 3.2.2. Thank you, @Yikun .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Yikun edited a comment on pull request #35595: [SPARK-38272][K8S][TESTS] Use `docker-desktop` instead of `docker-for-desktop` for Docker K8S IT deployMode and context name

Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35595:
URL: https://github.com/apache/spark/pull/35595#issuecomment-1047769036


   @dongjoon-hyun Did you mean this? : )
   
   Current fix keep the compatiblity of `deployMode, `make `docker-for-desktop` and `docker-desktop` point to `docker-desktop` backend.
   
   And drop the docker version support before 2.4.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org