You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/18 04:47:53 UTC

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31849: [SPARK-34757][CORE][DEPLOY] Ignore cache for SNAPSHOT dependencies in spark-submit

HyukjinKwon commented on a change in pull request #31849:
URL: https://github.com/apache/spark/pull/31849#discussion_r596549816



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -1153,6 +1153,8 @@ private[spark] object SparkSubmitUtils extends Logging {
     // We need a chain resolver if we want to check multiple repositories
     val cr = new ChainResolver
     cr.setName("spark-list")
+    cr.setChangingMatcher(PatternMatcher.REGEXP)
+    cr.setChangingPattern(".*-SNAPSHOT")

Review comment:
       While I agree with the intention (I faced the same problem and had to manually remove cache too), I concern that it will actually make users more confused in a way because it's Maven or Ivy's standard behaviour, and now we're changing how they work by default (which probably users don't know).
   
   I know it's very unlikely but some users might want to use one cached snapshot (presumably as they know the behaviours of Maven or Ivy resolvers work). Let's say, one CI regularly publishes snapshot, and users want to test one specific version created at the specific time. After this PR, they are forced to use the newest snapshot always.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org