You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by KaiXinXiaoLei <gi...@git.apache.org> on 2015/04/07 09:13:00 UTC
[GitHub] spark pull request: [CORE] EventLog load many times
GitHub user KaiXinXiaoLei opened a pull request:
https://github.com/apache/spark/pull/5386
[CORE] EventLog load many times
For example:
2015-04-07 23:04:32,406 | INFO | [log-replay-executor-0] | Replaying log path: hdfs://hacluster/sparkJobHistory/application_1428400855101_0004 | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-07 23:04:32,503 | INFO | [log-replay-executor-0] | Application log application_1428400855101_0004 loaded successfully. | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-07 23:04:42,410 | INFO | [log-replay-executor-0] | Replaying log path: hdfs://hacluster/sparkJobHistory/application_1428400855101_0004 | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-07 23:04:42,501 | INFO | [log-replay-executor-0] | Application log application_1428400855101_0004 loaded successfully. | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-07 23:04:52,406 | INFO | [log-replay-executor-0] | Replaying log path: hdfs://hacluster/sparkJobHistory/application_1428400855101_0004 | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-07 23:04:52,502 | INFO | [log-replay-executor-0] | Application log application_1428400855101_0004 loaded successfully. | org.apache.spark.Logging$class.logInfo(Logging.scala:59)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/KaiXinXiaoLei/spark eventlogLoad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5386.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5386
----
commit 4e9449f87ab3312987af6b646214611f74960ec6
Author: KaiXinXiaoLei <hu...@huawei.com>
Date: 2015-04-07T07:09:04Z
EventLog load many times
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by KaiXinXiaoLei <gi...@git.apache.org>.
Github user KaiXinXiaoLei commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-91093099
ok. thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90770559
Makes sense. @KaiXinXiaoLei do you mind closing this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90536855
[Test build #29790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29790/consoleFull) for PR 5386 at commit [`4e9449f`](https://github.com/apache/spark/commit/4e9449f87ab3312987af6b646214611f74960ec6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90518280
CC @vanzin as a follow up to SPARK-5582. Seems correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90518310
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90519483
[Test build #29790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29790/consoleFull) for PR 5386 at commit [`4e9449f`](https://github.com/apache/spark/commit/4e9449f87ab3312987af6b646214611f74960ec6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90771080
BTW, it's not impossible to fix, but the current fix is not correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] EventLog load many times
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90412194
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90663256
(Ah, btw, the tests fail because of what I explained above...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90643719
No, this is not correct. This is sub-optimal but necessary. If there is a new application that is added between polls but somehow happens to have the same mod time as the last poll, this patch would make that application be missed.
The current code is a little paranoid, but at least it's correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-90536869
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29790/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by KaiXinXiaoLei <gi...@git.apache.org>.
Github user KaiXinXiaoLei commented on the pull request:
https://github.com/apache/spark/pull/5386#issuecomment-91093107
ok. thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [CORE] The newest log will be loaded many time...
Posted by KaiXinXiaoLei <gi...@git.apache.org>.
Github user KaiXinXiaoLei closed the pull request at:
https://github.com/apache/spark/pull/5386
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org