You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/02 23:20:25 UTC
[GitHub] [spark] Gaurangi94 opened a new pull request #29630: SPARK-32097: Enable Spark History Server to read from multiple directories
Gaurangi94 opened a new pull request #29630:
URL: https://github.com/apache/spark/pull/29630
Change-Id: Ie3e2a6cc08b4c0e8770417c3b66f2f747bebefda
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
Currently logDir just refers to one directory. We would like to add a capability to HistoryServer UI to read from multiple directories.
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
To view data from multiple clusters/environments on the same history server
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
No
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
By running existing test suites.
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686870943
I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster.
If the rationalization of SPARK-32097 and SPARK-32135 is to make SHS be cluster-wise, then probably the concept of "clusters" needs to be defined and introduced, instead of having some fixes for workaround.
Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise and supports multi-clusters, I would prefer "isolated view" per cluster, like selecting the cluster first, and see restricted view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in SPARK-32135. So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Gaurangi94 commented on a change in pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
Gaurangi94 commented on a change in pull request #29630:
URL: https://github.com/apache/spark/pull/29630#discussion_r483157778
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -471,7 +471,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
val newLastScanTime = clock.getTimeMillis()
logDebug(s"Scanning $logDir with lastScanTime==$lastScanTime")
- val updated = Option(fs.listStatus(new Path(logDir))).map(_.toSeq).getOrElse(Nil)
+ val updated = Option(fs.globStatus(new Path(logDir + "/*"))).map(_.toSeq).getOrElse(Nil)
Review comment:
Yes, we are using a single file system in this case. This feature is useful when using external file systems for log data because then multiple directories will correspond to multiple clusters.
We could have a list too. Looking into that option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Gaurangi94 commented on a change in pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
Gaurangi94 commented on a change in pull request #29630:
URL: https://github.com/apache/spark/pull/29630#discussion_r485046482
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -471,7 +471,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
val newLastScanTime = clock.getTimeMillis()
logDebug(s"Scanning $logDir with lastScanTime==$lastScanTime")
- val updated = Option(fs.listStatus(new Path(logDir))).map(_.toSeq).getOrElse(Nil)
+ val updated = Option(fs.globStatus(new Path(logDir + "/*"))).map(_.toSeq).getOrElse(Nil)
Review comment:
The reason we chose to go with glob-pattern is that our service creates short-lived YARN clusters on which Spark applications are run. Log data goes to directories in a remote file-system. Since the number of these clusters can be really large, glob pattern would better fit our use-case, than a static list would.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #29630:
URL: https://github.com/apache/spark/pull/29630#discussion_r482809167
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -471,7 +471,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
val newLastScanTime = clock.getTimeMillis()
logDebug(s"Scanning $logDir with lastScanTime==$lastScanTime")
- val updated = Option(fs.listStatus(new Path(logDir))).map(_.toSeq).getOrElse(Nil)
+ val updated = Option(fs.globStatus(new Path(logDir + "/*"))).map(_.toSeq).getOrElse(Nil)
.filter { entry => isAccessible(entry.getPath) }
.filter { entry => !isProcessing(entry.getPath) }
Review comment:
This is playing as a "lock" for the log file. You can see where the lock flag is accessed or/and modified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Gaurangi94 commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
Gaurangi94 commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686655786
> SHS isn't a read-only; there's also a cleanup phase, and load and cleanup phases are running in parallel. Please take a look at the SHS code in master branch thoughtfully.
Thanks. I will go over those phases and ensure there aren't any concurrency concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686870943
I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster.
If the rationalization of SPARK-32097 and SPARK-32135 is to make SHS be cluster-wise, then probably the concept of "clusters" needs to be defined and introduced, instead of having some fixes for workaround.
Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise, I would prefer "isolated view" per cluster, like selecting the cluster first, and see restricted view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in SPARK-32135. So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Gaurangi94 commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
Gaurangi94 commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-688991583
> I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster.
>
> If the rationalization of [SPARK-32097](https://issues.apache.org/jira/browse/SPARK-32097) and [SPARK-32135](https://issues.apache.org/jira/browse/SPARK-32135) is to make SHS be cluster-wise, then probably the concept of "cluster" needs to be defined and introduced, instead of having some fixes for workaround.
>
> Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise and supports multi-clusters, I would prefer "isolated view" per cluster, like selecting the cluster first, and see filtered view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in [SPARK-32135](https://issues.apache.org/jira/browse/SPARK-32135). So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
Hey, I have updated the JIRA with a more elaborate description of our use-case. Initially, I had tried to keep it general but I noticed that has caused a lot of unintended confusion. Apologies for that. Could you review our use-case again and we can discuss what you think would be a better way to support it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29630:
URL: https://github.com/apache/spark/pull/29630#discussion_r482980562
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -471,7 +471,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
val newLastScanTime = clock.getTimeMillis()
logDebug(s"Scanning $logDir with lastScanTime==$lastScanTime")
- val updated = Option(fs.listStatus(new Path(logDir))).map(_.toSeq).getOrElse(Nil)
+ val updated = Option(fs.globStatus(new Path(logDir + "/*"))).map(_.toSeq).getOrElse(Nil)
Review comment:
yeah I don't quite get this, you are requiring all the directories to be under a single directory, I guess that makes this logic easier, but at the same time why the restriction and not have a list? If we are going to support multiple directories and make sure it works I don't see the reason to have this restriction. What if people have multiple clusters writing to different HDFS filesystems for instance.
I agree with @HeartSaVioR if we are going to support multiple directories we need to have a thorough look at all the logic here to make sure no other problems. I guess in this case you are using a single filesystem?
I think we need to flush out more of the overall goals and design first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686870943
I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster.
If the rationalization of SPARK-32097 and SPARK-32135 is to make SHS be cluster-wise, then probably the concept of "cluster" needs to be defined and introduced, instead of having some fixes for workaround.
Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise and supports multi-clusters, I would prefer "isolated view" per cluster, like selecting the cluster first, and see filtered view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in SPARK-32135. So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29630: SPARK-32097: Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686088419
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #29630:
URL: https://github.com/apache/spark/pull/29630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29630: SPARK-32097: Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686088419
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Gaurangi94 commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
Gaurangi94 commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686316853
> I'm not sure your PR really deals with reading from multiple directories. The change is listing -> glob with *. Could you please elaborate what is the difference? The change also doesn't have any new unit tests verifying the changes.
>
> In general comment with the idea, having multiple root directories are still possible, but probably better to be just a static list (IMHO) instead of regex, as listing with glob pattern is known to be very slow.
>
> One thing I'm afraid of having multiple root directories is, SHS is already very complicated in point of thread-safety view even we only allow single root directory, and it may make things more complicated. I'm on the fence on doing this, until we are clear that this won't make SHS more complicated.
Thanks for your response! By multiple directories I meant that a regex could potentially match more than one directory. In case of external file system, glob pattern might be better considering we will have to make just one over the network call. Also, it will be easier for the user to specify just one setting, instead of multiple values. What do you think?
I will add the unit tests. Thanks for pointing out.
MHS will function only as a read only server. Can thread-safety be an issue in that case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29630: SPARK-32097: Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686088730
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #29630:
URL: https://github.com/apache/spark/pull/29630#discussion_r482810050
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
##########
@@ -755,24 +755,6 @@ class FsHistoryProviderSuite extends SparkFunSuite with Matchers with Logging {
}
}
- testRetry("provider reports error after FS leaves safe mode") {
Review comment:
Please don't remove the existing test unless you have strong reason to do so. If your change breaks existing test, you may need to explain the reason about necessity of discarding this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686344779
SHS isn't a read-only; there's also a cleanup phase, and load and cleanup phases are running in parallel. Please take a look at the SHS code in master branch thoughtfully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-687553724
cc @rednaxelafx @jiangxb1987
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-747802122
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686262943
I'm not sure your PR really deals with reading from multiple directories. The change is listing -> glob with *. Could you please elaborate what is the difference? The change also doesn't have any new unit tests verifying the changes.
In general comment with the idea, having multiple root directories are still possible, but probably better to be just a static list (IMHO) instead of regex, as listing with glob pattern is known to be very slow.
One thing I'm afraid of having multiple root directories is, SHS is already very complicated in point of thread-safety view even we only allow single root directory, and it may make things more complicated. I'm on the fence on doing this, until we are clear that this won't make SHS more complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686870943
I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster.
If the rationalization of SPARK-32097 and SPARK-32135 is to make SHS be cluster-wise, then probably the concept of "cluster" needs to be defined and introduced, instead of having some fixes for workaround.
Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise and supports multi-clusters, I would prefer "isolated view" per cluster, like selecting the cluster first, and see restricted view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in SPARK-32135. So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29630:
URL: https://github.com/apache/spark/pull/29630#issuecomment-686870943
I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. If the rationalization of SPARK-32097 and SPARK-32135 is to make SHS be cluster-wise, then probably the concept of "clusters" needs to be defined and introduced, instead of having some fixes for workaround.
Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise, I would prefer "isolated view" per cluster, like selecting the cluster first, and see restricted view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in SPARK-32135. So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org