You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/15 04:58:24 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
HeartSaVioR opened a new pull request #31187:
URL: https://github.com/apache/spark/pull/31187
### What changes were proposed in this pull request?
`FsHistoryProvider#checkForLogs` already has `FileStatus` when constructing `SingleFileEventLogFileReader`, and there is no need to get the `FileStatus` again when `SingleFileEventLogFileReader#fileSizeForLastIndex`.
### Why are the changes needed?
This can reduce a lot of rpc calls and improve the speed of the history server.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
exist ut
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760663594
**[Test build #134081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134081/testReport)** for PR 31187 at commit [`0bf8a09`](https://github.com/apache/spark/commit/0bf8a09b071cbad4a14257805890597e2f6b6e5c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760690502
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134081/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760896656
Merging this as Github Action passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760698590
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38667/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760689350
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38667/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760713231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38667/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760654824
It sounds a bit odd we still need Mima exclusion after #30814... I'll play a bit to see whether we can remove Mima exclusion again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-762354772
yeah, this would make quite the difference. If you can do anything to go from a list to an open without an extra HEAD then life is a lot, lot better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760690502
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134081/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31187:
URL: https://github.com/apache/spark/pull/31187#issuecomment-760713231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38667/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #31187:
URL: https://github.com/apache/spark/pull/31187
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org