You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by XuefengWu <gi...@git.apache.org> on 2014/11/17 06:36:41 UTC
[GitHub] spark pull request: set a Sleep Serializer for test shuffle wait t...
GitHub user XuefengWu opened a pull request:
https://github.com/apache/spark/pull/3300
set a Sleep Serializer for test shuffle wait time
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/XuefengWu/spark SPARK-2208-shuffle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3300.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3300
----
commit 50039e75e6fe2c4bba8355ecfed274444fac89e2
Author: Xuefeng Wu <xf...@thoughtworks.com>
Date: 2014-11-17T05:32:04Z
set a Sleep Serializer for test shuffle wait time
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63711293
[Test build #23627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23627/consoleFull) for PR 3300 at commit [`3e82413`](https://github.com/apache/spark/commit/3e824139497ad2d7eadf1cfde17b6a624784fb2a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63722639
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23627/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63722630
[Test build #23627 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23627/consoleFull) for PR 3300 at commit [`3e82413`](https://github.com/apache/spark/commit/3e824139497ad2d7eadf1cfde17b6a624784fb2a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SlowSerializer extends org.apache.spark.serializer.Serializer`
* `class SlowSerializerInstance extends org.apache.spark.serializer.SerializerInstance `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by XuefengWu <gi...@git.apache.org>.
Github user XuefengWu closed the pull request at:
https://github.com/apache/spark/pull/3300
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3300#discussion_r20424429
--- Diff: core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala ---
@@ -240,13 +252,12 @@ class SparkListenerSuite extends FunSuite with LocalSparkContext with Matchers
taskInfoMetrics.map(_._2.executorDeserializeTime),
stageInfo + " executorDeserializeTime")
- /* Test is disabled (SEE SPARK-2208)
--- End diff --
Might this be a little more straightforward to just introduce a delay here, and/or loop with max number of iterations, to poll for this condition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by XuefengWu <gi...@git.apache.org>.
Github user XuefengWu commented on a diff in the pull request:
https://github.com/apache/spark/pull/3300#discussion_r20433243
--- Diff: core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala ---
@@ -240,13 +252,12 @@ class SparkListenerSuite extends FunSuite with LocalSparkContext with Matchers
taskInfoMetrics.map(_._2.executorDeserializeTime),
stageInfo + " executorDeserializeTime")
- /* Test is disabled (SEE SPARK-2208)
--- End diff --
@srowen sorry, could you show the code? I do not understand where could introduce delay directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by XuefengWu <gi...@git.apache.org>.
Github user XuefengWu commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63732194
It looks not work in same cases, for `fast` is a related concept. I'm looking for other solution for this test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: set a Sleep Serializer for test shuffle wait t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63261981
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3300#issuecomment-63710621
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3300#discussion_r20443123
--- Diff: core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala ---
@@ -240,13 +252,12 @@ class SparkListenerSuite extends FunSuite with LocalSparkContext with Matchers
taskInfoMetrics.map(_._2.executorDeserializeTime),
stageInfo + " executorDeserializeTime")
- /* Test is disabled (SEE SPARK-2208)
--- End diff --
My fault, I see that the problem is not that the named RDD fails to exist, but that the fetch time is 0. I think it makes sense that fetch waiting time includes deserialization but your test would prove that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2208] set a Sleep Serializer for test s...
Posted by XuefengWu <gi...@git.apache.org>.
Github user XuefengWu commented on a diff in the pull request:
https://github.com/apache/spark/pull/3300#discussion_r20433234
--- Diff: core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala ---
@@ -240,13 +252,12 @@ class SparkListenerSuite extends FunSuite with LocalSparkContext with Matchers
taskInfoMetrics.map(_._2.executorDeserializeTime),
stageInfo + " executorDeserializeTime")
- /* Test is disabled (SEE SPARK-2208)
--- End diff --
@srowen sorry, could you show the code? I do not understand where could introduce delay directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org