You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/18 02:52:31 UTC
[GitHub] [spark] williamhyun opened a new pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
williamhyun opened a new pull request #29796:
URL: https://github.com/apache/spark/pull/29796
### What changes were proposed in this pull request?
```scala
- fs.isDirectory(hadoopPath)
+ fs.getFileStatus(hadoopPath).isDirectory
```
```scala
- fs.isFile(new Path(inProgressLog))
+ fs.getFileStatus(new Path(inProgressLog)).isFile
```
### Why are the changes needed?
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2-hive-2.3/1244/consoleFull
```
[warn] /home/jenkins/workspace/spark-master-test-sbt-hadoop-3.2-hive-2.3/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:815: method isFile in class FileSystem is deprecated: see corresponding Javadoc for more information.
[warn] if (!fs.isFile(new Path(inProgressLog))) {
```
```
[warn] /home/jenkins/workspace/spark-master-test-sbt-hadoop-3.2-hive-2.3/core/src/main/scala/org/apache/spark/SparkContext.scala:1884: method isDirectory in class FileSystem is deprecated: see corresponding Javadoc for more information.
[warn] if (fs.isDirectory(hadoopPath)) {
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694753939
Merged to master and branch-3.0 (to reduce potential conflicts)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490682171
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
I missed that. Sorry. In that case, the utility methods will be identical with the deprecated Hadoop 3.2 `isFile/isDirectory` themselves. I'll close my PR. Thank you for your feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490709194
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -812,11 +812,12 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
try {
// Fetch the entry first to avoid an RPC when it's already removed.
listing.read(classOf[LogInfo], inProgressLog)
- if (!fs.isFile(new Path(inProgressLog))) {
+ if (!fs.getFileStatus(new Path(inProgressLog)).isFile) {
listing.delete(classOf[LogInfo], inProgressLog)
}
} catch {
case _: NoSuchElementException =>
+ case _: FileNotFoundException =>
Review comment:
This is added for getFileStatus at line 815.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694618049
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490687461
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
For example, `fs.getFileStatus(hadoopPath).isDirectory` here looks valid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694753939
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29796:
URL: https://github.com/apache/spark/pull/29796
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694654345
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694617722
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694654345
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490687765
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -1192,7 +1192,7 @@ class HiveDDLSuite
expectedDBUri,
Map.empty))
// the database directory was created
- assert(fs.exists(dbPath) && fs.isDirectory(dbPath))
+ assert(fs.exists(dbPath) && fs.getFileStatus(dbPath).isDirectory)
Review comment:
this too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694661637
**[Test build #128854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128854/testReport)** for PR 29796 at commit [`71a0465`](https://github.com/apache/spark/commit/71a046515efd306f36e4894069abe234271b99d2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694618049
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694696812
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694658323
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694662809
@HyukjinKwon I narrowed downed my PR according to your advice. Could you take a look again, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694696876
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128853/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490710569
##########
File path: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala
##########
@@ -58,7 +58,7 @@ private[streaming] object HdfsUtils {
// If we are really unlucky, the file may be deleted as we're opening the stream.
// This can happen as clean up is performed by daemon threads that may be left over from
// previous runs.
- if (!dfs.isFile(dfsPath)) null else throw e
+ if (!dfs.getFileStatus(dfsPath).isFile) null else throw e
Review comment:
This is safe because of lines 53 and 55.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694618030
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694662074
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694659598
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694652478
Thank you @HyukjinKwon I'll narrow down the score of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694659598
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490710752
##########
File path: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala
##########
@@ -92,6 +92,10 @@ private[streaming] object HdfsUtils {
def checkFileExists(path: String, conf: Configuration): Boolean = {
val hdpPath = new Path(path)
val fs = getFileSystemForPath(hdpPath, conf)
- fs.isFile(hdpPath)
+ try {
+ fs.getFileStatus(hdpPath).isFile
+ } catch {
+ case _: FileNotFoundException => false
+ }
Review comment:
This is the same logic with the isFile method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694617722
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490687211
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
@williamhyun, I think they were deprecated to avoid calling redundant `getFileStatus` at [HADOOP-13321](https://issues.apache.org/jira/browse/HADOOP-13321):
> These methods have a habit of fostering inefficient call patterns in applications, resulting in multiple redundant getFileStatus calls.
You can have one util that takes both `FileStatus` or `path` for example.
If you'd like to take a more conservative approach, you can keep only some valid calls (with `fs.exists()` as an example).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490687211
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
@williamhyun, I think they were deprecated to avoid calling redundant `getFileStatus` at [HADOOP-13321](https://issues.apache.org/jira/browse/HADOOP-13321):
> These methods have a habit of fostering inefficient call patterns in applications, resulting in multiple redundant getFileStatus calls.
You can have one util that takes both `FileStatus` and `path` for example.
If you'd like to take more conservative, you can keep some valid calls (with `fs.exists()` as an example).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694662074
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694696488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490673471
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
@williamhyun, looks `fs.getFileStatus(hadoopPath).isDirectory` isn't the exact synonym of `fs.isDirectory`, e.g.)
```java
public boolean isDirectory(Path f) throws IOException {
try {
return getFileStatus(f).isDirectory();
} catch (FileNotFoundException e) {
return false; // f does not exist
}
}
```
I think you can create utility methods under `Utils.scala` to reuse.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490708937
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
This is safe because of `if (!fs.exists(hadoopPath)) {` at line 1881.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694661637
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694696812
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694659074
**[Test build #128853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128853/testReport)** for PR 29796 at commit [`1c7dbae`](https://github.com/apache/spark/commit/1c7dbaef34827935d0b30bf3d6d55bcbb3005143).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694654723
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694753939
Merged to master, branch-3.0 and branch-2.4 (to reduce potential conflicts)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490687648
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/EventLogFileWritersSuite.scala
##########
@@ -213,7 +213,7 @@ class SingleEventLogFileWriterSuite extends EventLogFileWritersSuite {
compressionCodecShortName)
val finalLogPath = new Path(logPath)
- assert(fileSystem.exists(finalLogPath) && fileSystem.isFile(finalLogPath))
+ assert(fileSystem.exists(finalLogPath) && fileSystem.getFileStatus(finalLogPath).isFile)
Review comment:
this looks valid too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29796:
URL: https://github.com/apache/spark/pull/29796#discussion_r490673471
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1881,7 +1881,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (!fs.exists(hadoopPath)) {
throw new FileNotFoundException(s"Jar ${path} not found")
}
- if (fs.isDirectory(hadoopPath)) {
+ if (fs.getFileStatus(hadoopPath).isDirectory) {
Review comment:
@williamhyun, looks `fs.getFileStatus(hadoopPath).isDirectory` isn't exact synonym of `fs.isDirectory`, e.g.)
```java
public boolean isDirectory(Path f) throws IOException {
try {
return getFileStatus(f).isDirectory();
} catch (FileNotFoundException e) {
return false; // f does not exist
}
}
```
I think you can create a utility methods under `Utils.scala` to reuse.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] williamhyun closed pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
williamhyun closed pull request #29796:
URL: https://github.com/apache/spark/pull/29796
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29796: [SPARK-32930][CORE] Replace deprecated isFile/isDirectory methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29796:
URL: https://github.com/apache/spark/pull/29796#issuecomment-694654723
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org