You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "vihangk1 (via GitHub)" <gi...@apache.org> on 2023/06/26 21:31:02 UTC
[GitHub] [spark] vihangk1 opened a new pull request, #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
vihangk1 opened a new pull request, #41749:
URL: https://github.com/apache/spark/pull/41749
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
The `CacheManager` refreshFileIndexIfNecessary logic checks if the fileIndex root paths starts with the input path. This is problematic if the input path and root path share the prefixes but the root path is not a subdirectory of the input path. In such cases, the CacheManager can unnecessarily refresh the fileIndex which can fail the query if it does not have access to the rootPath for that SparkSession.
### Why are the changes needed?
Fixes the bug where the queries on cached dataframe APIs can fail if the cached path shares prefix with the different path.
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
Unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1611737602
@cloud-fan Do you know why the github action is failing for me? I fails due to "Unhandled error: TypeError: Cannot read properties of undefined (reading 'head_sha')"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244355231
##########
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:
##########
@@ -1685,4 +1683,22 @@ class CachedTableSuite extends QueryTest with SQLTestUtils
}
}
}
+
+ test("SPARK-44199: isSubDirectory tests") {
Review Comment:
can we add a new `CacheManagerSuite` to hold this new test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1246047499
##########
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:
##########
@@ -20,12 +20,9 @@ package org.apache.spark.sql
import java.io.{File, FilenameFilter}
import java.nio.file.{Files, Paths}
import java.time.{Duration, LocalDateTime, Period}
-
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1242988181
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
I am not sure I understand. Can you give more details on the example above? I the `mytable/col=A` gets updated, why would a rootPath which is not a subdirectory of `mytable/col=A` need to be refreshed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244352135
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
+ .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory))
+ .exists(isSubDir(qualifiedPath, _))
if (needToRefresh) fileIndex.refresh()
needToRefresh
}
+ /**
+ * Checks if the given child path is a sub-directory of the given parent path.
+ * @param qualifiedPathChild: Fully qualified child path
+ * @param qualifiedPathParent: Fully qualified parent path.
+ * @return True if the child path is a sub-directory of the given parent path. Otherwise, false.
+ */
+ def isSubDir(qualifiedPathParent: Path, qualifiedPathChild: Path): Boolean = {
+ Iterator.iterate(qualifiedPathChild)(_.getParent)
+ .takeWhile(_ != null)
+ .exists(_.toString.equals(qualifiedPathParent.toString))
Review Comment:
can't we rely on `Path.equals`?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
+ .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory))
+ .exists(isSubDir(qualifiedPath, _))
if (needToRefresh) fileIndex.refresh()
needToRefresh
}
+ /**
+ * Checks if the given child path is a sub-directory of the given parent path.
+ * @param qualifiedPathChild: Fully qualified child path
+ * @param qualifiedPathParent: Fully qualified parent path.
+ * @return True if the child path is a sub-directory of the given parent path. Otherwise, false.
+ */
+ def isSubDir(qualifiedPathParent: Path, qualifiedPathChild: Path): Boolean = {
+ Iterator.iterate(qualifiedPathChild)(_.getParent)
+ .takeWhile(_ != null)
+ .exists(_.toString.equals(qualifiedPathParent.toString))
Review Comment:
can't we rely on `Path#equals`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "Yikun (via GitHub)" <gi...@apache.org>.
Yikun commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1612347269
This might related to github incidents: https://www.githubstatus.com/incidents/ws2l6203cs7g , retry should help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1614878537
The tests `ExpressionImplUtilsSuite` and `RandomDataGeneratorSuite` look flaky. They work for me locally. Let me rebase the PR and retrigger the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1612381893
> This might related to github incidents: https://www.githubstatus.com/incidents/ws2l6203cs7g , retry should help.
Thanks for checking. The problem was with my permissions to the github actions on my fork. I think it should work now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1242983499
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
I think it'd break Hive's partitioned table. Think about that a partition like `mytable/col=A`, and `col` gets updated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1250156980
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
Actually yeah, I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244505929
##########
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:
##########
@@ -1685,4 +1683,22 @@ class CachedTableSuite extends QueryTest with SQLTestUtils
}
}
}
+
+ test("SPARK-44199: isSubDirectory tests") {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1619242649
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1242988181
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
I am not sure I understand. Can you give more details on the example above? If the `mytable/col=A` gets updated, why would a rootPath which is not a subdirectory of `mytable/col=A` need to be refreshed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #41749:
URL: https://github.com/apache/spark/pull/41749#issuecomment-1611965789
@HyukjinKwon @Yikun do you know why github action doesn't work here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244350726
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
If `col` gets updated, then we need to fresh all subdirectories of `mytable/col=A`. I think this bug fix is correct that subdirectory is not substring of the path string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vihangk1 commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "vihangk1 (via GitHub)" <gi...@apache.org>.
vihangk1 commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1244503861
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
+ .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory))
+ .exists(isSubDir(qualifiedPath, _))
if (needToRefresh) fileIndex.refresh()
needToRefresh
}
+ /**
+ * Checks if the given child path is a sub-directory of the given parent path.
+ * @param qualifiedPathChild: Fully qualified child path
+ * @param qualifiedPathParent: Fully qualified parent path.
+ * @return True if the child path is a sub-directory of the given parent path. Otherwise, false.
+ */
+ def isSubDir(qualifiedPathParent: Path, qualifiedPathChild: Path): Boolean = {
+ Iterator.iterate(qualifiedPathChild)(_.getParent)
+ .takeWhile(_ != null)
+ .exists(_.toString.equals(qualifiedPathParent.toString))
Review Comment:
I think we can. Let me change this. to use `Path.equals`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1245664275
##########
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:
##########
@@ -20,12 +20,9 @@ package org.apache.spark.sql
import java.io.{File, FilenameFilter}
import java.nio.file.{Files, Paths}
import java.time.{Duration, LocalDateTime, Period}
-
Review Comment:
let's revert the unnecessary changes in this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41749:
URL: https://github.com/apache/spark/pull/41749#discussion_r1242984023
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -366,14 +366,25 @@ class CacheManager extends Logging with AdaptiveSparkPlanHelper {
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path): Boolean = {
- val prefixToInvalidate = qualifiedPath.toString
val needToRefresh = fileIndex.rootPaths
- .map(_.makeQualified(fs.getUri, fs.getWorkingDirectory).toString)
- .exists(_.startsWith(prefixToInvalidate))
Review Comment:
It'd be good to add some comments about this while we're here in any event. It took me a lot of time to review ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #41749: [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
URL: https://github.com/apache/spark/pull/41749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org