You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/26 07:21:41 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
HeartSaVioR opened a new pull request #28930:
URL: https://github.com/apache/spark/pull/28930
### What changes were proposed in this pull request?
This patch fixes the missed spot - the test initializes FileStreamSinkLog with its "output" directory instead of "metadata" directory, hence the verification against sink log was no-op.
### Why are the changes needed?
Without the fix, the verification against sink log was no-op.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Checked with debugger in test, and verified `allFiles()` returns non-zero entries. (It returned zero entry, as there's no metadata.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446774988
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
+ FileStreamSink.checkEscapedMetadataPath(fs, metadataDir, sqlConf)
Review comment:
this is new code. So this PR fixes a bug instead of just fixing a test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650022532
**[Test build #124532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124532/testReport)** for PR 28930 at commit [`f7e6b4e`](https://github.com/apache/spark/commit/f7e6b4ebfd83fcdd18734abd15b9deca95ed6f52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446775390
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
+ FileStreamSink.checkEscapedMetadataPath(fs, metadataDir, sqlConf)
Review comment:
Oh sorry I should have explained before. That's to avoid test to copy same code (logic) what FileStreamSink does, so that we don't break the test when we somehow change it. So that's a refactor and doesn't mean there's a bug in FileStreamSink.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-651164306
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446775390
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
+ FileStreamSink.checkEscapedMetadataPath(fs, metadataDir, sqlConf)
Review comment:
Oh sorry I should have explained before. That's to avoid test to copy same code (logic) what FileStreamSink does, so that we don't break the test when we somehow change it. Doesn't mean there's a bug in FileStreamSink.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650022532
**[Test build #124532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124532/testReport)** for PR 28930 at commit [`f7e6b4e`](https://github.com/apache/spark/commit/f7e6b4ebfd83fcdd18734abd15b9deca95ed6f52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-651628933
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650981088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650023032
cc. @gengliangwang @cloud-fan @HyukjinKwon as they've reviewed #26671
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446775390
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
+ FileStreamSink.checkEscapedMetadataPath(fs, metadataDir, sqlConf)
Review comment:
Oh sorry I should have explained before. That's to avoid test to have same code what FileStreamSink does, so that we don't break the test when we somehow change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #28930:
URL: https://github.com/apache/spark/pull/28930
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650981088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650986588
**[Test build #124629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124629/testReport)** for PR 28930 at commit [`8f949a1`](https://github.com/apache/spark/commit/8f949a1c615c2c8824dd027e52cdb7f26d085c3f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446775390
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
+ FileStreamSink.checkEscapedMetadataPath(fs, metadataDir, sqlConf)
Review comment:
Oh sorry I should have explained before. That's to avoid test to copy same code (logic) what FileStreamSink does, so that we don't break the test when we somehow change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650023556
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650934435
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-651164306
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650023556
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650091356
I am not very familiar with streaming but LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-651141330
**[Test build #124629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124629/testReport)** for PR 28930 at commit [`8f949a1`](https://github.com/apache/spark/commit/8f949a1c615c2c8824dd027e52cdb7f26d085c3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650139180
**[Test build #124532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124532/testReport)** for PR 28930 at commit [`f7e6b4e`](https://github.com/apache/spark/commit/f7e6b4ebfd83fcdd18734abd15b9deca95ed6f52).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650986588
**[Test build #124629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124629/testReport)** for PR 28930 at commit [`8f949a1`](https://github.com/apache/spark/commit/8f949a1c615c2c8824dd027e52cdb7f26d085c3f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650140121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-651633651
Thanks all for reviewing and merging!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28930:
URL: https://github.com/apache/spark/pull/28930#discussion_r446774699
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
##########
@@ -55,6 +54,13 @@ object FileStreamSink extends Logging {
}
}
+ def getMetadataLogPath(path: Path, hadoopConf: Configuration, sqlConf: SQLConf): Path = {
+ val metadataDir = new Path(path, FileStreamSink.metadataDir)
+ val fs = metadataDir.getFileSystem(hadoopConf)
Review comment:
shall we pass the `fs` as a parameter as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650126259
Ah that's OK. Thanks for reviewing :) cc @dongjoon-hyun @zsxwing and @gatorsmile from #26639 (original PR of SPARK-29999)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28930:
URL: https://github.com/apache/spark/pull/28930#issuecomment-650140121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org