[GitHub] [spark] cloud-fan commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily

["cloud-fan (via GitHub)" <gi...@apache.org>]

cloud-fan commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244350726


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
       fileIndex: FileIndex,
       fs: FileSystem,
       qualifiedPath: Path): Boolean = {
-    val prefixToInvalidate = qualifiedPath.toString
     val needToRefresh = fileIndex.rootPaths
-      .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
-      .exists(_.startsWith(prefixToInvalidate))

Review Comment:
   If `col` gets updated, then we need to fresh all subdirectories of `mytable/col=A`. I think this bug fix is correct that subdirectory is not substring of the path string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org