You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ilganeli <gi...@git.apache.org> on 2015/03/27 22:17:06 UTC
[GitHub] spark pull request: [SPARK-5931][CORE][WIP] Use consistent naming ...
GitHub user ilganeli opened a pull request:
https://github.com/apache/spark/pull/5236
[SPARK-5931][CORE][WIP] Use consistent naming for time properties
I've added new utility methods to do the conversion from times specified as 120s, 240ms, 360us to convert to a consistent internal representation and I'm in the process of updating usage of these constants throughout the code to be consistent.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilganeli/spark SPARK-5931
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5236.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5236
----
commit 59bf9e1d590e49f451aaa7feeb5165572f205358
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:15:26Z
[SPARK-5931] Updated Utils and JavaUtils classes to add helper methods to handle time strings. Updated time strings in a few places to properly parse time
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88282370
I added the call into Java - this way we also don't need a separate test since it's all calling the same code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28004868
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,68 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length() - 1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ break;
+ }
+ }
--- End diff --
This whole method is really dense. Should we do something like the following instead:
```
private static long parseTimeString(String str, TimeUnit unit) {
String lower = str.toLowerCase().trim();
long parsedTime = -1;
TimeUnit parsedSuffix = null;
for (String suffix : timeSuffixes.keySet()) {
if (lower.endsWith(suffix)) {
try {
// e.g. "50ms" = 50, "3 min" = 3, "3a50s" = error
parsedTime = Long.parseLong(str.substring(0, str.length() - suffix.length()).trim());
if (parsedTime >= 0) {
parsedSuffix = timeSuffixes.get(suffix)
break;
}
} catch(NumberFormatException e) {
// continue
}
}
}
// No valid time suffix found
if (parsedSuffix == null) {
throw new SomeException(...)
}
return unit.convert(parsedTime, parsedSuffix)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88251869
[Test build #29502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29502/consoleFull) for PR 5236 at commit [`39164f9`](https://github.com/apache/spark/commit/39164f984800e7474215b1803b594c12def13ae6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87753337
[Test build #29413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29413/consoleFull) for PR 5236 at commit [`68f4e93`](https://github.com/apache/spark/commit/68f4e937a17d0f70440a2fb6c415f73de215033e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87111796
[Test build #29325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29325/consoleFull) for PR 5236 at commit [`59bf9e1`](https://github.com/apache/spark/commit/59bf9e1d590e49f451aaa7feeb5165572f205358).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87980593
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29466/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92532459
Ok, LGTM I'm merging this into master. It's great to see things on your todo list getting ticked off by others in the community. Thanks @ilganeli @srowen @vanzin!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88288471
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29508/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88181743
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29490/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792368
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
--- End diff --
This can just be part of the `catch` block and/or a direct part of the exception constructor? I was surprised to see the error message it as the first statement in the method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91052728
[Test build #29892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29892/consoleFull) for PR 5236 at commit [`7d19cdd`](https://github.com/apache/spark/commit/7d19cdd16fbe9914148331d542a2d636797a83a3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90993678
[Test build #29875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29875/consoleFull) for PR 5236 at commit [`d4efd26`](https://github.com/apache/spark/commit/d4efd269f2a8716a46254b87b17a31eb07ec61b4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88276490
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29504/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792094
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,15 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs",
+ s"${Utils.timeStringAsS(sc.conf.get("spark.network.timeout", "120s"))}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val checkTimeoutIntervalMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerTimeoutIntervalMs",
--- End diff --
Am I reading this wrong or does this flip the order of priority for these two properties? Looks like `spark.network.timeoutInterval` should take precedence? but now it's the fallback default. If that's so, this might be worth unpacking a bit with intermediate values anyway as it's a little hard to read the logic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91036561
[Test build #29889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29889/consoleFull) for PR 5236 at commit [`6f651a8`](https://github.com/apache/spark/commit/6f651a8ab68c5ceb00a22ce5b68eff5bc51e631b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001923
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
--- End diff --
perhaps we should accept both `m` and `min` for minutes. If `1d` or `1h` works then I would naturally expect `1m` to work as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87980583
[Test build #29466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29466/consoleFull) for PR 5236 at commit [`647b5ac`](https://github.com/apache/spark/commit/647b5ac0ba7dc846abcf46245d0d1a9f512decc0).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91878732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30079/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28263954
--- Diff: docs/configuration.md ---
@@ -429,10 +441,10 @@ Apart from these, the following properties are also available, and may be useful
</tr>
<tr>
<td><code>spark.shuffle.io.retryWait</code></td>
- <td>5</td>
+ <td>5s</td>
<td>
- (Netty only) Seconds to wait between retries of fetches. The maximum delay caused by retrying
- is simply <code>maxRetries * retryWait</code>, by default 15 seconds.
+ (Netty only) How long to wait between retries of fetches. The maximum delay caused by retrying
+ is simply <code>maxRetries * retryWait</code>, by default 5 seconds.
--- End diff --
Got it - I've fixed the wording to be clearer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92453159
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30176/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27806736
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if (!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ throw new NumberFormatException(timeError + "\n" + e.getMessage());
+ }
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ public static long timeStringAsUs(String str) throws NumberFormatException {
--- End diff --
I just wanted to add the additional error message clarifying appropriate suffixes which is why I handle the error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792206
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -77,12 +77,14 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
override def onStart() {
// Periodically revive offers to allow delay scheduling to work
- val reviveInterval = conf.getLong("spark.scheduler.revive.interval", 1000)
+ val reviveIntervalMs = Utils.timeStringAsMs(
+ conf.get("spark.scheduler.revive.interval", "1000ms"))
--- End diff --
I suppose you can say "1s" but it's not that important
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261212
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1011,6 +1013,22 @@ private[spark] object Utils extends Logging {
}
/**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in ms.
+ */
+ def timeStringAsMs(str: String) : Long = {
+ JavaUtils.timeStringAsMs(str)
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in seconds.
+ */
+ def timeStringAsSec(str: String) : Long = {
--- End diff --
no space before `:`, here and L1019
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27459792
--- Diff: docs/configuration.md ---
@@ -209,10 +211,10 @@ Apart from these, the following properties are also available, and may be useful
<td>(none)</td>
<td>
Set a special library path to use when launching the driver JVM.
-
+
--- End diff --
I would turn it off. The whitespace itself doesn't matter at all functionally in most contexts. But it can create more noise in the diff. More important it can create spurious merge conflicts. An instance here or there doesn't matter much but I actively try to not make whitespace changes for this reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88254053
[Test build #29503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29503/consoleFull) for PR 5236 at commit [`39164f9`](https://github.com/apache/spark/commit/39164f984800e7474215b1803b594c12def13ae6).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90195392
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29747/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92514243
Looking strong. I think that's all the comments addressed now. Going once, going twice?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515898
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
+ val timeError = "Time must be specified as seconds (s), " +
--- End diff --
nit: indentation is wrong
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-97313146
Hah. Thanks.
Sent with Good (www.good.com)
-----Original Message-----
From: Reynold Xin [notifications@github.com<ma...@github.com>]
Sent: Wednesday, April 29, 2015 01:36 AM Eastern Standard Time
To: apache/spark
Cc: Ganelin, Ilya
Subject: Re: [spark] [SPARK-5931][CORE] Use consistent naming for time properties (#5236)
I'm late to the party, but just want to say -- this is super cool!
—
Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/5236#issuecomment-97312212>.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-89059698
@andrewor14 Thanks man - let me know if it needs anything else. I didn't end up changing any of the names for existing configs or introducing new versions of those but I think that's ok for time parameters because there's only a couple with explicit suffixes. I think it could be done but it's a bit of a pain in the neck and would complicate the code for ingesting the modified parameters (since we'd now need to check for both A and B). SPARK-5932 would be worse since there's many more fields with explicit suffixes there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28266570
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +125,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix;
+ long val;
+ Matcher m = Pattern.compile("(-?[0-9]+)([a-z]+)?").matcher(lower);
--- End diff --
ok, that's fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517486
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
--- End diff --
I think this method would be simpler if it took the desired "default" unit as an argument. For example:
def convert(str: String, tgt: TimeUnit): Long = {
timeSuffixes
.collectFirst { case (k,v) if str.endsWith(k) =>
tgt.convert(str.substring(0, str.length() - k.length()).toLong, v)
}
.getOrElse(str.toLong)
}
Then:
def timeStringAsMs(str : String) : Long = convert(str, TimeUnit.MILLISECONDS)
(Assuming I understood what it means to have the time string with no suffix.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91010124
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29879/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88288460
[Test build #29508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29508/consoleFull) for PR 5236 at commit [`dd0a680`](https://github.com/apache/spark/commit/dd0a68027ab2924128638d903d5ad2d8dcb35bdf).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441010
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
+ private def timeStringToUs(str: String) : Long = {
+ try {
+ val lower = str.toLowerCase.trim()
+ if (lower.endsWith("ms")) {
--- End diff --
I'd add at least `min`, `h` and `d` as valid suffixes here. Also, something like this might be easier to code for:
val suffixes = Map(
"us" -> 1,
"ms" -> 1000,
"s" -> 1000 * 1000,
"min" -> 60 * 1000 * 1000,
"h" -> 60 * 60 * 1000 * 1000,
// etc
Then you can iterate over the map instead of having cascading if / else checks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28010527
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,68 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length() - 1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ break;
+ }
+ }
--- End diff --
yeah, do that!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28266158
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +125,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix;
+ long val;
+ Matcher m = Pattern.compile("(-?[0-9]+)([a-z]+)?").matcher(lower);
--- End diff --
Hi Andrew - that's correct and intentional. I think this is in line with other configs at the moment, e.g. I don't believe that ```1 g``` works when specifying how much memory to allocate for Spark's usage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27519974
--- Diff: core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala ---
@@ -76,7 +76,7 @@ private[spark] object MetadataCleanerType extends Enumeration {
// initialization of StreamingContext. It's okay for users trying to configure stuff themselves.
private[spark] object MetadataCleaner {
def getDelaySeconds(conf: SparkConf): Int = {
- conf.getInt("spark.cleaner.ttl", -1)
+ Utils.timeStringAsS(conf.get("spark.cleaner.ttl", "-1s")).toInt
--- End diff --
I can just make it -1 as it was before. The function will assume it's a seconds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91014706
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29875/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28250502
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
-
+ private val slaveTimeoutMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerSlaveTimeoutMs", "120s")
+ private val executorTimeoutMs =
+ sc.conf.getTimeAsSec("spark.network.timeout", s"${slaveTimeoutMs}ms") * 1000
+
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val timeoutIntervalMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerTimeoutIntervalMs", "60s")
+ private val checkTimeoutIntervalMs =
+ sc.conf.getTimeAsSec("spark.network.timeoutInterval", s"${timeoutIntervalMs}ms") * 1000
--- End diff --
Sean. It can't because then the default unit is assumed to be ms when it's really seconds.
Sent with Good (www.good.com)
-----Original Message-----
From: Sean Owen [notifications@github.com<ma...@github.com>]
Sent: Saturday, April 11, 2015 12:51 PM Eastern Standard Time
To: apache/spark
Cc: Ganelin, Ilya
Subject: Re: [spark] [SPARK-5931][CORE] Use consistent naming for time properties (#5236)
In core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala<https://github.com/apache/spark/pull/5236#discussion_r28196499>:
> // "spark.network.timeoutInterval" uses "seconds", while
> // "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
> - private val checkTimeoutIntervalMs =
> - sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
> - getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
> + private val timeoutIntervalMs =
> + sc.conf.getTimeAsMs("spark.storage.blockManagerTimeoutIntervalMs", "60s")
> + private val checkTimeoutIntervalMs =
> + sc.conf.getTimeAsSec("spark.network.timeoutInterval", s"${timeoutIntervalMs}ms") * 1000
Same, can go straight to ms?
—
Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/5236/files#r28196499>.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90163754
[Test build #29747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29747/consoleFull) for PR 5236 at commit [`ff40bfe`](https://github.com/apache/spark/commit/ff40bfe6aaf17b17d334060800025d1442437ca5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91044958
[Test build #29889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29889/consoleFull) for PR 5236 at commit [`6f651a8`](https://github.com/apache/spark/commit/6f651a8ab68c5ceb00a22ce5b68eff5bc51e631b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27547205
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ public static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if(!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
--- End diff --
I can't initialize it to null because we have a call to suffix.length(). If it's not set, we'll get a NullPointerException. I think using the containsKey is fine, it's fairly clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28287888
--- Diff: docs/running-on-yarn.md ---
@@ -48,9 +48,9 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.am.waitTime</code></td>
- <td>100000</td>
+ <td>100s</td>
--- End diff --
(either that or we leave out that line L51 in `configuration.md`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90195367
[Test build #29747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29747/consoleFull) for PR 5236 at commit [`ff40bfe`](https://github.com/apache/spark/commit/ff40bfe6aaf17b17d334060800025d1442437ca5).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91010746
[Test build #29880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29880/consoleFull) for PR 5236 at commit [`cbd2ca6`](https://github.com/apache/spark/commit/cbd2ca68b54df440372fcdea64a0ffa4ddca40c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196841
--- Diff: docs/configuration.md ---
@@ -732,14 +745,14 @@ Apart from these, the following properties are also available, and may be useful
</tr>
<tr>
<td><code>spark.executor.heartbeatInterval</code></td>
- <td>10000</td>
+ <td>10000ms</td>
<td>Interval (milliseconds) between each executor's heartbeats to the driver. Heartbeats let
--- End diff --
If the units are now specified in the value, "(millseconds)" can be removed? and is it better to write 10s?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87777829
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29413/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88018848
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29467/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515740
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +611,9 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs = Utils.timeStringAsS(conf.get("spark.files.fetchTimeout","60s")).toInt*1000
--- End diff --
nit: space after `,`, spaces around `*`
Why not `timeStringAsMs`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87473474
If the two in the JIRA are all there are, then yea that's not much to worry about, probably. I still think that it's not worth _adding_ a suffix to existing properties though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88304640
[Test build #29510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29510/consoleFull) for PR 5236 at commit [`bf779b0`](https://github.com/apache/spark/commit/bf779b0e348e599e6b98ceb267a4b30a7ee8ed5a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88162431
[Test build #29490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29490/consoleFull) for PR 5236 at commit [`1858197`](https://github.com/apache/spark/commit/1858197cbb7f0fb75337d25092a525a2aa70f785).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28000092
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -80,16 +80,16 @@ private[spark] class ExecutorAllocationManager(
Integer.MAX_VALUE)
// How long there must be backlogged tasks for before an addition is triggered (seconds)
- private val schedulerBacklogTimeout = conf.getLong(
- "spark.dynamicAllocation.schedulerBacklogTimeout", 5)
+ private val schedulerBacklogTimeoutS = Utils.timeStringAsS(conf.get(
--- End diff --
very minor, but in general I find it weird for variables to have a capital letter at the end. I would call this either `schedulerBacklogTimeoutSeconds` or just leave it as before. Not a big deal if you don't change this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90137379
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29743/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87112397
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29329/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87163930
[Test build #29339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29339/consoleFull) for PR 5236 at commit [`8f741e1`](https://github.com/apache/spark/commit/8f741e19f66b011decb3cca285606a733a08990b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87761781
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29409/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91119705
[Test build #29924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29924/consoleFull) for PR 5236 at commit [`642a06d`](https://github.com/apache/spark/commit/642a06db778b40da8364122137c5c671814a46b5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87117196
[Test build #29326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29326/consoleFull) for PR 5236 at commit [`4933fda`](https://github.com/apache/spark/commit/4933fda30ce468d72e646536489468e295800775).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92530750
[Test build #30204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30204/consoleFull) for PR 5236 at commit [`4526c81`](https://github.com/apache/spark/commit/4526c817fd68bc24b7534d4fed285fea84b2a2fa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92547091
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30204/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
GitHub user ilganeli reopened a pull request:
https://github.com/apache/spark/pull/5236
[SPARK-5931][CORE] Use consistent naming for time properties
I've added new utility methods to do the conversion from times specified as e.g. 120s, 240ms, 360us to convert to a consistent internal representation. I've updated usage of these constants throughout the code to be consistent.
I believe I've captured all usages of time-based properties throughout the code. I've also updated variable names in a number of places to reflect their units for clarity and updated documentation where appropriate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilganeli/spark SPARK-5931
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5236.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5236
----
commit 59bf9e1d590e49f451aaa7feeb5165572f205358
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:15:26Z
[SPARK-5931] Updated Utils and JavaUtils classes to add helper methods to handle time strings. Updated time strings in a few places to properly parse time
commit 404f8c3762d7e23fdf3ef8d158d68f5b5aa38360
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:25:15Z
Updated usage of spark.core.connection.ack.wait.timeout
commit 7db6d2a73ea4de5bb52bcdf8e4af7edcc06a10dd
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:27:57Z
Updated usage of spark.akka.timeout
commit 4933fda30ce468d72e646536489468e295800775
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:30:05Z
Updated usage of spark.storage.blockManagerSlaveTimeout
commit c9f5cad6a35cc2f405d874a5ca62fa3f92ef0a8b
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:39:15Z
Updated spark.shuffle.io.retryWait
commit 21ef3dd6165022a1edc396c50817641d3321b1f7
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:40:52Z
updated spark.shuffle.sasl.timeout
commit 064ebd67faf655f03d0e02763f53f1df60b3e891
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:50:51Z
Updated usage of spark.cleaner.ttl
commit 7320c87289ded3f61607070a4098d00dc5ee17be
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T21:56:23Z
updated spark.akka.heartbeat.interval
commit 272c2159195948ccae9b399126f8e85015e1aaae
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:00:57Z
Updated spark.locality.wait
commit 3352d34d925650fdba34b05fc3521e15d18fcd11
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:02:59Z
Updated spark.scheduler.maxRegisteredResourcesWaitingTime
commit 3f1cfc84e669df7c2198ed21d776fd391977f34f
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:05:04Z
Updated spark.scheduler.revive.interval
commit 6d1518e8870dc91fec86d75be0cac11d3e1dc2b4
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:07:57Z
Upated spark.speculation.interval
commit 2fcc91c7cd062ca67c5b9e1565d13533e7013c2f
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:13:51Z
Updated spark.dynamicAllocation.executorIdleTimeout
commit 5181597688a6d6f0d9be311784582d033c908225
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:17:58Z
Updated spark.dynamicAllocation.schedulerBacklogTimeout
commit c6a00954f33994c35e9d76cb0222b35ce46bab69
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:22:49Z
Updated spark.core.connection.auth.wait.timeout
commit cde9bffef6ded083a9dfa80486687ddd07e85640
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:25:44Z
Updated spark.streaming.blockInterval
commit 42477aadc1f39911d325a0c703fae2ad55a6c42a
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T22:28:32Z
Updated configuration doc with note on specifying time properties
commit 9a29d8d3f502f43c237859e9f75f5d6c76662915
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-03-27T23:16:58Z
Fixed misuse of time in streaming context test
commit 34f87c26dd936305fec3d00d6a8cc1e6bc48734c
Author: Ilya Ganelin <il...@gmail.com>
Date: 2015-03-28T05:27:58Z
Update Utils.scala
Fixed ordering to fix overlap of s with ms and us.
commit 8f741e19f66b011decb3cca285606a733a08990b
Author: Ilya Ganelin <il...@gmail.com>
Date: 2015-03-28T05:28:50Z
Update JavaUtils.java
Fixed overlap of (s) with (ms) and (us) in cascading if.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88230631
Looks OK to me, left some minor comments and suggestions.
Could you also add unit tests to `UtilsSuite.scala`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92452711
[Test build #30176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30176/consoleFull) for PR 5236 at commit [`951ca2d`](https://github.com/apache/spark/commit/951ca2d150e2d0b046573d65bebcb0e15e158c36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91032518
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29880/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001272
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +613,9 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs = Utils.timeStringAsMs(conf.get("spark.files.fetchTimeout", "60s")).toInt
--- End diff --
I think we should just keep the lowest granularity at seconds for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87774103
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27806074
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +613,9 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs = Utils.timeStringAsMs(conf.get("spark.files.fetchTimeout", "60s")).toInt
--- End diff --
I wanted to leave it as 1min for clarity for that reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90229595
[Test build #29748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29748/consoleFull) for PR 5236 at commit [`1465390`](https://github.com/apache/spark/commit/14653907e118ead01ec3b0fe6640b5fcaba6d199).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28262232
--- Diff: docs/configuration.md ---
@@ -429,10 +441,10 @@ Apart from these, the following properties are also available, and may be useful
</tr>
<tr>
<td><code>spark.shuffle.io.retryWait</code></td>
- <td>5</td>
+ <td>5s</td>
<td>
- (Netty only) Seconds to wait between retries of fetches. The maximum delay caused by retrying
- is simply <code>maxRetries * retryWait</code>, by default 15 seconds.
+ (Netty only) How long to wait between retries of fetches. The maximum delay caused by retrying
+ is simply <code>maxRetries * retryWait</code>, by default 5 seconds.
--- End diff --
wrong! I filed a JIRA for this specifically because the wording here is confusing. In fact, this was fixed in #4769 but somehow the changes there got lost? @srowen are you aware of this? I suspect the author of #3863 accidentally dropped your changes when resolving merge conflicts.
In any case, this should remain 15 seconds (this refers to the product, not the value of the config itself).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90129220
[Test build #29743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29743/consoleFull) for PR 5236 at commit [`6387772`](https://github.com/apache/spark/commit/638777293c1049d0b68020868a6aec4b4573d58d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88184173
**[Test build #29485 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29485/consoleFull)** for PR 5236 at commit [`bac9edf`](https://github.com/apache/spark/commit/bac9edfe3feb285dc33b2212534841cfe859515f) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE][WIP] Use consistent naming ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87097725
[Test build #29326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29326/consoleFull) for PR 5236 at commit [`4933fda`](https://github.com/apache/spark/commit/4933fda30ce468d72e646536489468e295800775).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91009958
[Test build #29879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29879/consoleFull) for PR 5236 at commit [`1a1122c`](https://github.com/apache/spark/commit/1a1122c80d966d650690d1a1e271a9846a9fec08).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515772
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
--- End diff --
nit: add empty line before comment start
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515951
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us."
+
+ try {
+ val lower = str.toLowerCase.trim()
+ var suffix: String = ""
+ timeSuffixes.foreach(s => {
+ if(lower.endsWith(s._1)) {
--- End diff --
nit: space after `if`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261494
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +36,64 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
+
+ test("timeConversion") {
+ // Test -1
+ assert(Utils.timeStringAsSec("-1") === -1)
+
+ // Test zero
+ assert(Utils.timeStringAsSec("0") === 0)
+
+ assert(Utils.timeStringAsSec("1") === 1)
+ assert(Utils.timeStringAsSec("1s") === 1)
+ assert(Utils.timeStringAsSec("1000ms") === 1)
+ assert(Utils.timeStringAsSec("1000000us") === 1)
+ assert(Utils.timeStringAsSec("1min") === TimeUnit.MINUTES.toSeconds(1))
--- End diff --
can you add the same test for `1m` as well, same in L59
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28265326
--- Diff: docs/running-on-yarn.md ---
@@ -48,7 +48,7 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.am.waitTime</code></td>
- <td>100000</td>
+ <td>100000ms</td>
<td>
In yarn-cluster mode, time in milliseconds for the application master to wait for the
--- End diff --
you missed one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28265566
--- Diff: docs/configuration.md ---
@@ -35,8 +35,20 @@ val conf = new SparkConf()
val sc = new SparkContext(conf)
{% endhighlight %}
-Note that we can have more than 1 thread in local mode, and in cases like spark streaming, we may actually
-require one to prevent any sort of starvation issues.
+Note that we can have more than 1 thread in local mode, and in cases like Spark Streaming, we may
+actually require one to prevent any sort of starvation issues.
+
+Properties that specify some time duration should be configured with a unit of time.
+The following format is accepted:
+
+ 5s (seconds)
+ 25ms (milliseconds)
--- End diff --
should these two be switched? Otherwise the ordering here isn't from smallest to largest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27845205
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala ---
@@ -176,7 +177,7 @@ class StreamingContextSuite extends FunSuite with BeforeAndAfter with Timeouts w
test("stop gracefully") {
val conf = new SparkConf().setMaster(master).setAppName(appName)
- conf.set("spark.cleaner.ttl", "3600")
+ conf.set("spark.cleaner.ttl", "3600ms")
--- End diff --
They were always seconds before, so this test meant "3600 seconds" before. 1 hour = 3600 seconds sounds like the intent rather than 3.6 seconds = 3600ms. I imagine the point of this is "a long TTL"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91048824
There is. The first test for every case has no suffix.
Sent with Good (www.good.com)
-----Original Message-----
From: Marcelo Vanzin [notifications@github.com<ma...@github.com>]
Sent: Wednesday, April 08, 2015 05:53 PM Eastern Standard Time
To: apache/spark
Cc: Ganelin, Ilya
Subject: Re: [spark] [SPARK-5931][CORE] Use consistent naming for time properties (#5236)
@ilganeli<https://github.com/ilganeli> you should add a unit test for that. It seems there are no unit tests for suffix-less strings.
—
Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/5236#issuecomment-91046918>.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90726746
Andrew - if you pass the string 5000 it will be interpreted as 5000ms as intended. I use 5s as shorthand in the code for brevity. I also believe blockTimeoutMs is the only one that's explicitly named with units. Other ones are not, so deprecating just that field shouldn't be difficult.
Sent with Good (www.good.com)
-----Original Message-----
From: andrewor14 [notifications@github.com<ma...@github.com>]
Sent: Tuesday, April 07, 2015 03:56 PM Eastern Standard Time
To: apache/spark
Cc: Ganelin, Ilya
Subject: Re: [spark] [SPARK-5931][CORE] Use consistent naming for time properties (#5236)
@ilganeli<https://github.com/ilganeli> I took a quick look at this and I believe this does not currently handle backward compatibility. For instance, spark.storage.blockManagerSlaveTimeoutMs used to take in the string "5000", but now I have to do "5s" otherwise it will throw an exception. The bigger issue is that it's very strange to see a config ending with Ms to accept a value "5s" or "20ms". I think this patch does only half of what I intended, and the other half must accompany this half.
However, to deprecate all these configs is currently a very verbose process. This is why I opened SPARK-5933<https://issues.apache.org/jira/browse/SPARK-5933>, which I actually see as a prerequisite before we solve either SPARK-5931 or SPARK-5932. For these reasons I would suggest that we put this PR on hold for now. I definitely do want to merge some variant of the changes here, but I don't think there's currently a great way forward yet.
—
Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/5236#issuecomment-90713134>.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28265905
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +125,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix;
+ long val;
+ Matcher m = Pattern.compile("(-?[0-9]+)([a-z]+)?").matcher(lower);
--- End diff --
this doesn't handle whitespace does it? e.g. `1 min`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91060457
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29899/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28224908
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
-
+ private val slaveTimeoutMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerSlaveTimeoutMs", "120s")
+ private val executorTimeoutMs =
+ sc.conf.getTimeAsSec("spark.network.timeout", s"${slaveTimeoutMs}ms") * 1000
--- End diff --
Right, that's why. Ignore all the similar comments below then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91056511
[Test build #29899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29899/consoleFull) for PR 5236 at commit [`69fedcc`](https://github.com/apache/spark/commit/69fedccd2fcd1916caf5fcf0bb08e9986b4fe2cf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792240
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -79,7 +80,8 @@ private[spark] object AkkaUtils extends Logging {
val logAkkaConfig = if (conf.getBoolean("spark.akka.logAkkaConfig", false)) "on" else "off"
val akkaHeartBeatPauses = conf.getInt("spark.akka.heartbeat.pauses", 6000)
--- End diff --
This is also a time property in seconds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28002150
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length()-1;
--- End diff --
space around `-`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28005633
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,68 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length() - 1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ break;
+ }
+ }
--- End diff --
Ugh, that was much longer than expected. It would have been a lot more concise in Scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27524619
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
--- End diff --
I see - you're relying on the order of the map to resolve conflicts. You could still have a one-liner for that but it would be a little less efficient, so the latest code is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88384432
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29524/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515408
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -52,17 +52,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, scheduler: TaskSchedule
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val networkTimeoutS = Utils.timeStringAsS(sc.conf.get("spark.network.timeout","120s"))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs", s"${networkTimeoutS}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val networkTimeoutIntervalS =
+ Utils.timeStringAsS(sc.conf.get("spark.network.timeoutInterval","60s"))
+ private val checkTimeoutIntervalMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerTimeoutIntervalMs", s"${networkTimeoutIntervalS}s"))
private var timeoutCheckingTask: Cancellable = null
-
--- End diff --
please leave this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27530579
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +35,32 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
-
+ test("timeConversion") {
+ assert(Utils.timeStringAsS("1") === 1)
+ assert(Utils.timeStringAsS("1s") === 1)
+ assert(Utils.timeStringAsS("1000ms") === 1)
+ assert(Utils.timeStringAsS("1000000us") === 1)
+ assert(Utils.timeStringAsS("1min") === 60)
+ assert(Utils.timeStringAsS("1h") === 60 * 60)
+ assert(Utils.timeStringAsS("1d") === 60 * 60 * 24)
+
+ assert(Utils.timeStringAsMs("1") === 1)
+ assert(Utils.timeStringAsMs("1ms") === 1)
+ assert(Utils.timeStringAsMs("1000us") === 1)
+ assert(Utils.timeStringAsMs("1s") === 1000)
+ assert(Utils.timeStringAsMs("1min") === 1000 * 60)
+ assert(Utils.timeStringAsMs("1h") === 1000 * 60 * 60)
+ assert(Utils.timeStringAsMs("1d") === 1000 * 60 * 60 * 24)
--- End diff --
minor: you could use `TimeUnit.DAYS.toMillis(1)` here, for example (also in other tests).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517850
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +121,59 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long timeStringToUs(String str) throws NumberFormatException {
--- End diff --
This method can be very similar to the Scala one (using an `com.google.common.collect.ImmutableMap` to hold the valid suffixes). Maybe that's what you meant by updating the Java code later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27530590
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +35,32 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
-
--- End diff --
nit: keep this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90620239
[Test build #29797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29797/consoleFull) for PR 5236 at commit [`cbf41db`](https://github.com/apache/spark/commit/cbf41db9e33e6be592668a2d9b263f1fa40111c8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88252557
[Test build #29502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29502/consoleFull) for PR 5236 at commit [`39164f9`](https://github.com/apache/spark/commit/39164f984800e7474215b1803b594c12def13ae6).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88283549
[Test build #29508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29508/consoleFull) for PR 5236 at commit [`dd0a680`](https://github.com/apache/spark/commit/dd0a68027ab2924128638d903d5ad2d8dcb35bdf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87185772
This change actually stops supporting a bunch of existing config properties, right? I don't think we can do that.
As discussed in https://issues.apache.org/jira/browse/SPARK-3859 it's possible to deprecate the existing ones, add new ones, and support both. In practice, the deprecated configs will stick around a long time. (cf. MapReduce and it's "deprecated" properties from like 20 years ago).
I personally am not sure about changing the config properties just for this as a result. However I can see changing some internal variable names to reflect their units.
CC @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27806115
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,15 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs",
+ s"${Utils.timeStringAsS(sc.conf.get("spark.network.timeout", "120s"))}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val checkTimeoutIntervalMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerTimeoutIntervalMs",
--- End diff --
Hm but the block manager timeout now has precedence. The network prop did before. Right? I can't see how this isnt different. Or else I'm really blind now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792523
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala ---
@@ -176,7 +177,7 @@ class StreamingContextSuite extends FunSuite with BeforeAndAfter with Timeouts w
test("stop gracefully") {
val conf = new SparkConf().setMaster(master).setAppName(appName)
- conf.set("spark.cleaner.ttl", "3600")
+ conf.set("spark.cleaner.ttl", "3600ms")
--- End diff --
The units are seconds here rather than ms -- here and in the next diff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92462131
[Test build #30184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30184/consoleFull) for PR 5236 at commit [`de3bff9`](https://github.com/apache/spark/commit/de3bff9995f108cdd6183a4fe36b4c5781d861c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91130309
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29924/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87108631
[Test build #29330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29330/consoleFull) for PR 5236 at commit [`42477aa`](https://github.com/apache/spark/commit/42477aadc1f39911d325a0c703fae2ad55a6c42a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441147
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
--- End diff --
This is not used anywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28002559
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length()-1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
--- End diff --
should we just `break` after we find this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001133
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -848,14 +848,14 @@ private[spark] class TaskSetManager(
}
private def getLocalityWait(level: TaskLocality.TaskLocality): Long = {
- val defaultWait = conf.get("spark.locality.wait", "3000")
+ val defaultWait = conf.get("spark.locality.wait", "3s")
level match {
case TaskLocality.PROCESS_LOCAL =>
- conf.get("spark.locality.wait.process", defaultWait).toLong
+ Utils.timeStringAsMs(conf.get("spark.locality.wait.process", defaultWait))
case TaskLocality.NODE_LOCAL =>
- conf.get("spark.locality.wait.node", defaultWait).toLong
+ Utils.timeStringAsMs(conf.get("spark.locality.wait.node", defaultWait))
case TaskLocality.RACK_LOCAL =>
- conf.get("spark.locality.wait.rack", defaultWait).toLong
+ Utils.timeStringAsMs(conf.get("spark.locality.wait.rack", defaultWait))
case _ => 0L
}
--- End diff --
just an idea, maybe we can rewrite this as:
```
val defaultWait = conf.get("spark.locality.wait", "3s")
val localityWaitKey =
key match {
case TaskLocality.PROCESS_LOCAL => "spark.locality.wait.process"
case TaskLocality.NODE_LOCAL => "spark.locality.wait.node"
case TaskLocality.RACK_LOCAL => "spark.locality.wait.rack"
}
Utils.timeStringAsMs(conf.get(localityWaitKey, defaultWait)
```
Looks nicer IMO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90713134
@ilganeli I took a quick look at this and I believe this does not currently handle backward compatibility. For instance, `spark.storage.blockManagerSlaveTimeoutMs` used to take in the string "5000", but now I have to do "5s" otherwise it will throw an exception. The bigger issue is that it's very strange to see a config ending with `Ms` to accept a value "5s" or "20ms". I think this patch does only half of what I intended, and the other half must accompany this half.
However, to deprecate all these configs is currently a very verbose process. This is why I opened [SPARK-5933](https://issues.apache.org/jira/browse/SPARK-5933), which I actually see as a prerequisite before we solve either SPARK-5931 or SPARK-5932. For these reasons I would suggest that we put this PR on hold for now. I definitely do want to merge some variant of the changes here, but I don't think there's currently a great way forward yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27530673
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,64 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ new ImmutableMap.Builder<String, TimeUnit>()
--- End diff --
nit: indentation.
Generally I've seen people use `ImmutableMap.<String, TimeUnit>builder()` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792494
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -132,7 +132,7 @@ class ReceiverSuite extends TestSuiteBase with Timeouts with Serializable {
test("block generator") {
val blockGeneratorListener = new FakeBlockGeneratorListener
val blockInterval = 200
--- End diff --
Same, I suppose you could consider renaming this to have "MS" on the end for consistency. The self-documenting name is great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001996
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length()-1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+
+ throw new NumberFormatException(timeError + "\n" + e.getMessage());
+ }
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ public static long timeStringAsUs(String str) {
+ return parseTimeString(str, TimeUnit.MICROSECONDS);
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to milliseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in ms.
+ */
+ public static long timeStringAsMs(String str) {
+ return parseTimeString(str, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to seconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in seconds.
+ */
+ public static long timeStringAsS(String str) {
--- End diff --
can we call this `timeStringAsSeconds`? (also `Utils.timeStringAsSeconds`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92493085
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30184/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28288002
--- Diff: docs/running-on-yarn.md ---
@@ -48,9 +48,9 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.am.waitTime</code></td>
- <td>100000</td>
+ <td>100s</td>
--- End diff --
Andrew - I'd prefer to leave out that line. I think it makes things simpler and encourages transitioning to the new paradigm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91130301
[Test build #29924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29924/consoleFull) for PR 5236 at commit [`642a06d`](https://github.com/apache/spark/commit/642a06db778b40da8364122137c5c671814a46b5).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517554
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us."
+
+ try {
+ val lower = str.toLowerCase.trim()
+ var suffix: String = ""
+ timeSuffixes.foreach(s => {
+ if(lower.endsWith(s._1)) {
+ suffix = s._1
+ }
+ })
+
+ (timeSuffixes.get(suffix), str.substring(0, str.length - suffix.length).toLong)
+ } catch {
+
--- End diff --
nit: empty line not needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91010123
[Test build #29879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29879/consoleFull) for PR 5236 at commit [`1a1122c`](https://github.com/apache/spark/commit/1a1122c80d966d650690d1a1e271a9846a9fec08).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87745427
[Test build #29411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29411/consoleFull) for PR 5236 at commit [`3a12dd8`](https://github.com/apache/spark/commit/3a12dd8c85d936fb74298f92f8a55951998ce450).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE][WIP] Use consistent naming ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87102754
[Test build #29328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29328/consoleFull) for PR 5236 at commit [`064ebd6`](https://github.com/apache/spark/commit/064ebd67faf655f03d0e02763f53f1df60b3e891).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196816
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1011,6 +1014,30 @@ private[spark] object Utils extends Logging {
}
/**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ def timeStringAsUs(str: String): Long = {
--- End diff --
Are the microseconds methods used anywhere? I'm not aware of anything that uses this resolution, but might have missed it. There aren't us methods in SparkConf.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91049694
> There is. The first test for every case has no suffix.
Ah, my bad. In that case the previous code was fine. (just tried and `HashMap.containsKey(null)` works.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88018831
**[Test build #29467 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29467/consoleFull)** for PR 5236 at commit [`1c0c07c`](https://github.com/apache/spark/commit/1c0c07c5f59cf9e163bde6d04a8c6dedb10517bf) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88254061
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29503/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27542107
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ public static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if(!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
--- End diff --
nit: I'd initialize `suffix` to `null` and check for that here instead of calling `containsKey`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88367369
Last batch of comments incorporated. Thanks for the thorough review, Marcelo!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27807200
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala ---
@@ -176,7 +177,7 @@ class StreamingContextSuite extends FunSuite with BeforeAndAfter with Timeouts w
test("stop gracefully") {
val conf = new SparkConf().setMaster(master).setAppName(appName)
- conf.set("spark.cleaner.ttl", "3600")
+ conf.set("spark.cleaner.ttl", "3600ms")
--- End diff --
Sean - I know the units for ```spark.cleaner.ttl``` are typically seconds, however for the sake of this test, if we set it to 3600 seconds nothing will ever happen. I think in this test it's expected to be in milliseconds, or am I mistaken?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792453
--- Diff: network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -68,7 +71,9 @@ public int numConnectionsPerPeer() {
public int sendBuf() { return conf.getInt("spark.shuffle.io.sendBuffer", -1); }
/** Timeout for a single round trip of SASL token exchange, in milliseconds. */
- public int saslRTTimeoutMs() { return conf.getInt("spark.shuffle.sasl.timeout", 30) * 1000; }
+ public int saslRTTimeoutMs() {
+ return (int) JavaUtils.timeStringAsS(conf.get("spark.shuffle.sasl.timeout", "30s")) * 1000;
--- End diff --
Same here and in the next change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88277010
I wouldn't mind calling into the Java for this. I think it pays to avoid code duplication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517867
--- Diff: network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -37,8 +37,11 @@ public boolean preferDirectBufs() {
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
- int defaultTimeout = conf.getInt("spark.network.timeout", 120);
- return conf.getInt("spark.shuffle.io.connectionTimeout", defaultTimeout) * 1000;
+ long defaultNetworkTimeoutS = JavaUtils.timeStringAsS(
+ conf.get("spark.network.timeout","120s"));
--- End diff --
nit: space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28000506
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val networkTimeoutS = Utils.timeStringAsS(sc.conf.get("spark.network.timeout", "120s"))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs",
+ s"${networkTimeoutS}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val timeoutIntervalS = Utils.timeStringAsS(sc.conf
+ .get("spark.network.timeoutInterval", "60s"))
+ private val checkTimeoutIntervalMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerTimeoutIntervalMs",
+ s"${timeoutIntervalS}s"))
--- End diff --
same here, this changes behavior
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792468
--- Diff: network/shuffle/src/test/java/org/apache/spark/network/shuffle/RetryingBlockFetcherSuite.java ---
@@ -54,7 +54,7 @@
@Before
public void beforeEach() {
System.setProperty("spark.shuffle.io.maxRetries", "2");
- System.setProperty("spark.shuffle.io.retryWait", "0");
+ System.setProperty("spark.shuffle.io.retryWait", "0s");
--- End diff --
No need to change this, but as an observation, "0" ought to be fine everywhere without units since it's unambiguous.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27845096
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if (!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ throw new NumberFormatException(timeError + "\n" + e.getMessage());
+ }
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ public static long timeStringAsUs(String str) throws NumberFormatException {
--- End diff --
That's fine but I mean the method need not be declared as `throws NumberFormatException`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196496
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
-
+ private val slaveTimeoutMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerSlaveTimeoutMs", "120s")
+ private val executorTimeoutMs =
+ sc.conf.getTimeAsSec("spark.network.timeout", s"${slaveTimeoutMs}ms") * 1000
--- End diff --
Why not just `getTimeAsMs` here without the `* 1000`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001323
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1011,6 +1013,30 @@ private[spark] object Utils extends Logging {
}
/**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ def timeStringAsUs(str: String): Long = {
+ JavaUtils.timeStringAsUs(str)
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in ms.
+ */
+ def timeStringAsMs(str : String) : Long = {
--- End diff --
no space before `:`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28002033
--- Diff: network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -37,8 +37,11 @@ public boolean preferDirectBufs() {
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
- int defaultTimeout = conf.getInt("spark.network.timeout", 120);
- return conf.getInt("spark.shuffle.io.connectionTimeout", defaultTimeout) * 1000;
+ long defaultNetworkTimeoutS = JavaUtils.timeStringAsS(
+ conf.get("spark.network.timeout", "120s"));
+ long defaultTimeoutMs = JavaUtils.timeStringAsMs(
+ conf.get("spark.shuffle.io.connectionTimeout", defaultNetworkTimeoutS + "s"));
--- End diff --
should be 2 indents only on this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92185204
All changes pulled in - I think all parties should be happy now :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87799003
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29417/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88276371
Some more nits, but looks good. It would be nice to have a `JavaUtilsSuite` - it sucks that this code has to exist in two different places, but it's the price to pay I guess. :-/
I wonder if people would be ok with `Utils.scala` calling into `JavaUtils`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27542162
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1011,6 +1013,30 @@ private[spark] object Utils extends Logging {
}
/**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ def timeStringAsUs(str: String): Long = {
+ JavaUtils.parseTimeString(str, TimeUnit.MICROSECONDS)
--- End diff --
nit: if you call the corresponding method in `JavaUtils`, then `parseTimeString` can be private and the code here becomes a little simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91704038
@andrewor14 Is this in a good state now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27409405
--- Diff: docs/configuration.md ---
@@ -209,10 +211,10 @@ Apart from these, the following properties are also available, and may be useful
<td>(none)</td>
<td>
Set a special library path to use when launching the driver JVM.
-
+
--- End diff --
Sean - to follow up on the white-space comment. The reason that there's white space changes is that there's a lot of trailing white space in these files which my IDE is stripping. I could disable white-space stripping but in this case, I think that this is a fix worth making since it shouldn't be there in the first place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92215221
[Test build #30138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30138/consoleFull) for PR 5236 at commit [`bc04e05`](https://github.com/apache/spark/commit/bc04e05a1e61dc9eb14949e4f51737ddab7e39f2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88324638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29516/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27530735
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,64 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ new ImmutableMap.Builder<String, TimeUnit>()
+ .put("us",TimeUnit.MICROSECONDS)
+ .put("ms",TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ suffix = tail;
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch(NumberFormatException e) {
--- End diff --
nit: space before `(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92457128
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30179/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88249310
Latests fixes merged. Unit tests added. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792450
--- Diff: network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -37,8 +37,11 @@ public boolean preferDirectBufs() {
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
- int defaultTimeout = conf.getInt("spark.network.timeout", 120);
- return conf.getInt("spark.shuffle.io.connectionTimeout", defaultTimeout) * 1000;
+ long defaultNetworkTimeoutS = JavaUtils.timeStringAsS(
+ conf.get("spark.network.timeout", "120s"));
+ long defaultTimeoutS = JavaUtils.timeStringAsS(
--- End diff --
Can this be parsed straight to milliseconds then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792288
--- Diff: core/src/test/scala/org/apache/spark/network/nio/ConnectionManagerSuite.scala ---
@@ -261,7 +261,7 @@ class ConnectionManagerSuite extends FunSuite {
val clientConf = new SparkConf
clientConf.set("spark.authenticate", "false")
val ackTimeout = 30
--- End diff --
You could consider renaming this `acktimeoutS` too, but not that important either way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196834
--- Diff: docs/configuration.md ---
@@ -35,8 +35,21 @@ val conf = new SparkConf()
val sc = new SparkContext(conf)
{% endhighlight %}
-Note that we can have more than 1 thread in local mode, and in cases like spark streaming, we may actually
-require one to prevent any sort of starvation issues.
+Note that we can have more than 1 thread in local mode, and in cases like spark streaming, we may
--- End diff --
spark streaming -> Spark Streaming. If in fact microseconds are removed, they can be removed in the doc below this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5236
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28000695
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -436,14 +436,14 @@ private[spark] class Executor(
* This thread stops running when the executor is stopped.
*/
private def startDriverHeartbeater(): Unit = {
- val interval = conf.getInt("spark.executor.heartbeatInterval", 10000)
+ val intervalMs = Utils.timeStringAsMs(conf.get("spark.executor.heartbeatInterval", "10s"))
val thread = new Thread() {
override def run() {
- // Sleep a random interval so the heartbeats don't end up in sync
- Thread.sleep(interval + (math.random * interval).asInstanceOf[Int])
+ // Sleep a random intervalMs so the heartbeats don't end up in sync
--- End diff --
I don't think the "interval" here was referring to the variable, so could you please revert this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92331633
LGTM. Thanks for your persistence. Let me leave this for another day for further comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92186301
[Test build #30138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30138/consoleFull) for PR 5236 at commit [`bc04e05`](https://github.com/apache/spark/commit/bc04e05a1e61dc9eb14949e4f51737ddab7e39f2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27542113
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ public static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if(!tail.equals("s") ||
--- End diff --
nit: space after `if`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87118346
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792433
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if (!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ throw new NumberFormatException(timeError + "\n" + e.getMessage());
+ }
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ public static long timeStringAsUs(String str) throws NumberFormatException {
--- End diff --
`NumberFormatException` is unchecked. You can put it in the javadoc but I think (?) common practice is not to declare unchecked exceptions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196909
--- Diff: network/common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -37,8 +37,11 @@ public boolean preferDirectBufs() {
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
- int defaultTimeout = conf.getInt("spark.network.timeout", 120);
- return conf.getInt("spark.shuffle.io.connectionTimeout", defaultTimeout) * 1000;
+ long defaultNetworkTimeoutS = JavaUtils.timeStringAsSec(
+ conf.get("spark.network.timeout", "120s"));
+ long defaultTimeoutMs = JavaUtils.timeStringAsSec(
--- End diff --
One more instance of "can this be parsed straight to ms"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196560
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -174,6 +174,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
getOption(key).getOrElse(defaultValue)
}
+ /** Get a time parameter as seconds; throws a NoSuchElementException if it's not set. If no
+ * suffix is provided then seconds are assumed.
+ */
+ def getTimeAsSec(key: String): Long = {
+ Utils.timeStringAsSec(getOption(key).getOrElse(throw new NoSuchElementException(key)))
--- End diff --
I think you can just call `get(key)` and rely on it to perform exactly this logic? (Am I missing some separate discussion we already had on this?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87117207
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29326/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28260990
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -848,16 +848,14 @@ private[spark] class TaskSetManager(
}
private def getLocalityWait(level: TaskLocality.TaskLocality): Long = {
- val defaultWait = conf.get("spark.locality.wait", "3000")
- level match {
- case TaskLocality.PROCESS_LOCAL =>
- conf.get("spark.locality.wait.process", defaultWait).toLong
- case TaskLocality.NODE_LOCAL =>
- conf.get("spark.locality.wait.node", defaultWait).toLong
- case TaskLocality.RACK_LOCAL =>
- conf.get("spark.locality.wait.rack", defaultWait).toLong
- case _ => 0L
- }
+ val defaultWait = conf.get("spark.locality.wait", "3s")
+ val localityWaitKey = level match {
+ case TaskLocality.PROCESS_LOCAL => "spark.locality.wait.process"
+ case TaskLocality.NODE_LOCAL => "spark.locality.wait.node"
+ case TaskLocality.RACK_LOCAL => "spark.locality.wait.rack"
+ case _ => ""
--- End diff --
this should be `return 0L` instead to match old behavior
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515219
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -52,17 +52,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, scheduler: TaskSchedule
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val networkTimeoutS = Utils.timeStringAsS(sc.conf.get("spark.network.timeout","120s"))
--- End diff --
nit: space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515380
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -52,17 +52,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, scheduler: TaskSchedule
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val networkTimeoutS = Utils.timeStringAsS(sc.conf.get("spark.network.timeout","120s"))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs", s"${networkTimeoutS}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val networkTimeoutIntervalS =
+ Utils.timeStringAsS(sc.conf.get("spark.network.timeoutInterval","60s"))
--- End diff --
nit: space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88254816
[Test build #29504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29504/consoleFull) for PR 5236 at commit [`b2fc965`](https://github.com/apache/spark/commit/b2fc9652c519254f889135196b809e59658637f7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28033893
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +125,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ long val = -1;
+ Matcher m = Pattern.compile("(-?[0-9]+)([a-z]+)?").matcher(lower);
+ if (m.matches()) {
+ val = Long.parseLong(m.group(1));
+ suffix = m.group(2);
+ } else {
+ throw new NumberFormatException("Failed to parse time string: " + str);
+ }
+
+ return unit.convert(val, (suffix != null) && timeSuffixes.containsKey(suffix) ?
--- End diff --
This will use the `unit` parameter for any suffix that is not in `timeSuffixes`. The `containsKey` check should be separate and an exception should be thrown if the suffix is not there.
The unit tests don't catch this because all the invalid suffixes have spaces in them, so the regex doesn't match.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91032508
[Test build #29880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29880/consoleFull) for PR 5236 at commit [`cbd2ca6`](https://github.com/apache/spark/commit/cbd2ca68b54df440372fcdea64a0ffa4ddca40c7).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261343
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -174,6 +174,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
getOption(key).getOrElse(defaultValue)
}
+ /** Get a time parameter as seconds; throws a NoSuchElementException if it's not set. If no
+ * suffix is provided then seconds are assumed.
+ */
--- End diff --
What's the difference in this context? this works as javadoc too. The formatting could be more like:
```
/**
* Docs...
* @throws
*/
```
but should still render OK either way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27515677
--- Diff: core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala ---
@@ -76,7 +76,7 @@ private[spark] object MetadataCleanerType extends Enumeration {
// initialization of StreamingContext. It's okay for users trying to configure stuff themselves.
private[spark] object MetadataCleaner {
def getDelaySeconds(conf: SparkConf): Int = {
- conf.getInt("spark.cleaner.ttl", -1)
+ Utils.timeStringAsS(conf.get("spark.cleaner.ttl", "-1s")).toInt
--- End diff --
`-1s` looks really weird. Instead, I'd to:
conf.getOption("spark.cleaner.ttl").map(Utils.timeStringAsS).getOrElse(-1)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792404
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if (!tail.equals("s") ||
--- End diff --
This is very unlikely to change, but I wonder if it's less brittle to check `endsWith` and that the preceding character exists and is a digit? instead of managing the overlap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91867393
[Test build #30079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30079/consoleFull) for PR 5236 at commit [`25d3f52`](https://github.com/apache/spark/commit/25d3f5228297ed60b9adc43689957b4586636320).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28000465
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val networkTimeoutS = Utils.timeStringAsS(sc.conf.get("spark.network.timeout", "120s"))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs",
+ s"${networkTimeoutS}s"))
--- End diff --
this doesn't preserve the order. Before `spark.network.timeout` takes higher precedence, and if that's not defined we default to `spark.storage.blockManagerSlaveTimeoutMs`, which is deprecated IIRC. Currently the priority is inverted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90229613
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29748/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28264329
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +36,64 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
+
+ test("timeConversion") {
+ // Test -1
+ assert(Utils.timeStringAsSec("-1") === -1)
+
+ // Test zero
+ assert(Utils.timeStringAsSec("0") === 0)
+
+ assert(Utils.timeStringAsSec("1") === 1)
+ assert(Utils.timeStringAsSec("1s") === 1)
+ assert(Utils.timeStringAsSec("1000ms") === 1)
+ assert(Utils.timeStringAsSec("1000000us") === 1)
+ assert(Utils.timeStringAsSec("1min") === TimeUnit.MINUTES.toSeconds(1))
+ assert(Utils.timeStringAsSec("1h") === TimeUnit.HOURS.toSeconds(1))
+ assert(Utils.timeStringAsSec("1d") === TimeUnit.DAYS.toSeconds(1))
+
+ assert(Utils.timeStringAsMs("1") === 1)
+ assert(Utils.timeStringAsMs("1ms") === 1)
+ assert(Utils.timeStringAsMs("1000us") === 1)
+ assert(Utils.timeStringAsMs("1s") === TimeUnit.SECONDS.toMillis(1))
+ assert(Utils.timeStringAsMs("1min") === TimeUnit.MINUTES.toMillis(1))
+ assert(Utils.timeStringAsMs("1h") === TimeUnit.HOURS.toMillis(1))
+ assert(Utils.timeStringAsMs("1d") === TimeUnit.DAYS.toMillis(1))
+
+ // Test invalid strings
+ try {
+ Utils.timeStringAsMs("This breaks 600s")
+ assert(false) // We should never reach this
+ } catch {
+ case e: NumberFormatException => assert(true)
+ }
--- End diff --
That is actually super cool. Thanks for pointing that out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88384404
[Test build #29524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29524/consoleFull) for PR 5236 at commit [`76cfa27`](https://github.com/apache/spark/commit/76cfa2799ceae93534710c28545465c0a2b970f2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88324632
[Test build #29516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29516/consoleFull) for PR 5236 at commit [`bf779b0`](https://github.com/apache/spark/commit/bf779b0e348e599e6b98ceb267a4b30a7ee8ed5a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517602
--- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -27,7 +27,7 @@ import org.scalatest.FunSuite
import org.apache.spark._
import org.apache.spark.executor.TaskMetrics
-import org.apache.spark.util.ManualClock
+import org.apache.spark.util.{Utils, ManualClock}
--- End diff --
nit: sort order
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-89907768
[Test build #29739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29739/consoleFull) for PR 5236 at commit [`5193d5f`](https://github.com/apache/spark/commit/5193d5fe6d36a4aac8fe8c2892177dd216b775ab).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli closed the pull request at:
https://github.com/apache/spark/pull/5236
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88160874
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87113176
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29330/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27542123
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,69 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ public static long parseTimeString(String str, TimeUnit unit) throws NumberFormatException {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
+ "E.g. 50s, 100ms, or 250us.";
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ if (lower.endsWith(tail)) {
+ // Handle special case for overlapping suffixes
+ if(!tail.equals("s") ||
+ (tail.equals("s") && !lower.endsWith("us") && !lower.endsWith("ms")))
+ {
+ suffix = tail;
+ }
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ throw new NumberFormatException(timeError + "\n" + e.toString());
--- End diff --
nit: `+ e.getMessage()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28001958
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,66 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length()-1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ }
+ }
+
+ return unit.convert(Long.parseLong(str.substring(0, str.length() - suffix.length())),
+ timeSuffixes.containsKey(suffix) ? timeSuffixes.get(suffix) : unit);
+ } catch (NumberFormatException e) {
+ String timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), microseconds (us), minutes (min) hour (h), or day(d). " +
--- End diff --
space before `(d)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441074
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
+ private def timeStringToUs(str: String) : Long = {
+ try {
+ val lower = str.toLowerCase.trim()
+ if (lower.endsWith("ms")) {
+ lower.substring(0, lower.length - 2).toLong * 1000
+ } else if (lower.endsWith("us")) {
+ lower.substring(0, lower.length - 2).toLong
+ } else if (lower.endsWith("s")) {
+ lower.substring(0, lower.length - 1).toLong * 1000 * 1000
+ } else {
+ // Invalid suffix, force correct formatting
+ lower.toLong
+ }
+ } catch {
+ case e: NumberFormatException => throw new NumberFormatException(timeError)
+ }
+ }
+
+ /**
+ * Convert a time parameter such as (50s, 100ms, or 250us) to microseconds for internal use. If
+ * no suffix is provided, the passed number is assumed to be in us.
+ */
+ @throws(classOf[NumberFormatException])
+ def timeStringAsUs(str: String): Long = {
+ timeStringToUs(str)
--- End diff --
nit: too much indentation (here and in other places too)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87775159
[Test build #29417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29417/consoleFull) for PR 5236 at commit [`70ac213`](https://github.com/apache/spark/commit/70ac2138d8c1c23bc3bbd4fa91607686b2c329cc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27521217
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,67 @@ private[spark] object Utils extends Logging {
)
}
+ val timeSuffixes = Map (
+ "us" -> TimeUnit.MICROSECONDS,
+ "ms" -> TimeUnit.MILLISECONDS,
+ "s" -> TimeUnit.SECONDS,
+ "min" -> TimeUnit.MINUTES,
+ "h" -> TimeUnit.HOURS,
+ "d" -> TimeUnit.DAYS
+ )
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ def parseTimeString(str: String) : (Option[TimeUnit], Long) = {
--- End diff --
Marcelo - this doesn't correctly handle the overlapping suffix case (e.g. 60us is treated as 60s). I do like using convert though. I've added that in. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27517947
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala ---
@@ -73,9 +73,9 @@ class StreamingContextSuite extends FunSuite with BeforeAndAfter with Timeouts w
test("from conf with settings") {
val myConf = SparkContext.updatedConf(new SparkConf(false), master, appName)
- myConf.set("spark.cleaner.ttl", "10")
+ myConf.set("spark.cleaner.ttl", "10s")
ssc = new StreamingContext(myConf, batchDuration)
- assert(ssc.conf.getInt("spark.cleaner.ttl", -1) === 10)
+ assert(Utils.timeStringAsS(ssc.conf.get("spark.cleaner.ttl", "-1s")) === 10)
--- End diff --
same comment about `-1s`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27805802
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,15 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
+ private val executorTimeoutMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerSlaveTimeoutMs",
+ s"${Utils.timeStringAsS(sc.conf.get("spark.network.timeout", "120s"))}s"))
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val checkTimeoutIntervalMs = Utils.timeStringAsMs(
+ sc.conf.get("spark.storage.blockManagerTimeoutIntervalMs",
--- End diff --
Sean - order of priorities is the same. I had introduced a temp variable previously that I was recommended to inline (which brought us back to here). I think I'll restore it for the sake of clarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91046918
@ilganeli you should add a unit test for that. It seems there are no unit tests for suffix-less strings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-89052524
Thanks for working on this @ilganeli. I will take a look at this in the next day or two.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196499
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
-
+ private val slaveTimeoutMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerSlaveTimeoutMs", "120s")
+ private val executorTimeoutMs =
+ sc.conf.getTimeAsSec("spark.network.timeout", s"${slaveTimeoutMs}ms") * 1000
+
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val timeoutIntervalMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerTimeoutIntervalMs", "60s")
+ private val checkTimeoutIntervalMs =
+ sc.conf.getTimeAsSec("spark.network.timeoutInterval", s"${timeoutIntervalMs}ms") * 1000
--- End diff --
Same, can go straight to ms?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91052741
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29892/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441234
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
+ private def timeStringToUs(str: String) : Long = {
+ try {
+ val lower = str.toLowerCase.trim()
+ if (lower.endsWith("ms")) {
+ lower.substring(0, lower.length - 2).toLong * 1000
+ } else if (lower.endsWith("us")) {
+ lower.substring(0, lower.length - 2).toLong
+ } else if (lower.endsWith("s")) {
+ lower.substring(0, lower.length - 1).toLong * 1000 * 1000
+ } else {
+ // Invalid suffix, force correct formatting
+ lower.toLong
+ }
+ } catch {
+ case e: NumberFormatException => throw new NumberFormatException(timeError)
--- End diff --
You should also propagate the original error. It might be the case that the error has nothing to do with the lack of a suffix (e.g. `1!11ms`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-89922473
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29739/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28006218
--- Diff: network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java ---
@@ -121,4 +123,68 @@ private static boolean isSymlink(File file) throws IOException {
}
return !fileInCanonicalDir.getCanonicalFile().equals(fileInCanonicalDir.getAbsoluteFile());
}
+
+ private static ImmutableMap<String, TimeUnit> timeSuffixes =
+ ImmutableMap.<String, TimeUnit>builder()
+ .put("us", TimeUnit.MICROSECONDS)
+ .put("ms", TimeUnit.MILLISECONDS)
+ .put("s", TimeUnit.SECONDS)
+ .put("m", TimeUnit.MINUTES)
+ .put("min", TimeUnit.MINUTES)
+ .put("h", TimeUnit.HOURS)
+ .put("d", TimeUnit.DAYS)
+ .build();
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a time count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ private static long parseTimeString(String str, TimeUnit unit) {
+ String lower = str.toLowerCase().trim();
+
+ try {
+ String suffix = "";
+ for (String tail: timeSuffixes.keySet()) {
+ int charIdx = lower.length() - tail.length() - 1;
+ if (lower.endsWith(tail) && (charIdx >= 0 && Character.isDigit(lower.charAt(charIdx)))) {
+ suffix = tail;
+ break;
+ }
+ }
--- End diff --
Using regexes this would be pretty short.
Matcher m = Pattern.compile("([0-9]+)([a-z]+)?");
if (m.matches()) {
long val = Long.parseLong(m.group(1));
String suffix = m.group(2);
}
Or something along those lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90768155
@vanzin @ilganeli Oh I see, I didn't realize that 5000 would still work. Then it seems fine. I plan to look at this in much closer detail today or tomorrow to make sure that we haven't missed any configs, but from first glance it seems that we need to at least fix "spark.yarn.scheduler.heartbeat.interval-ms" (from the JIRA). Maybe there are a few others.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28263636
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +36,64 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
+
+ test("timeConversion") {
+ // Test -1
+ assert(Utils.timeStringAsSec("-1") === -1)
--- End diff --
Negative values are used in one of the other test suites, specifically the ```StreamingContextSuite``` with ```spark.cleaner.ttl``` so we need to handle this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE][WIP] Use consistent naming ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87107949
[Test build #29329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29329/consoleFull) for PR 5236 at commit [`cde9bff`](https://github.com/apache/spark/commit/cde9bffef6ded083a9dfa80486687ddd07e85640).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28260637
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -174,6 +174,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
getOption(key).getOrElse(defaultValue)
}
+ /** Get a time parameter as seconds; throws a NoSuchElementException if it's not set. If no
+ * suffix is provided then seconds are assumed.
+ */
--- End diff --
style: in Spark we use java docs not scala docs. See other methods in this file for reference
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87113158
[Test build #29330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29330/consoleFull) for PR 5236 at commit [`42477aa`](https://github.com/apache/spark/commit/42477aadc1f39911d325a0c703fae2ad55a6c42a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28265238
--- Diff: docs/configuration.md ---
@@ -35,8 +35,20 @@ val conf = new SparkConf()
val sc = new SparkContext(conf)
{% endhighlight %}
-Note that we can have more than 1 thread in local mode, and in cases like spark streaming, we may actually
-require one to prevent any sort of starvation issues.
+Note that we can have more than 1 thread in local mode, and in cases like Spark Streaming, we may
+actually require one to prevent any sort of starvation issues.
+
+Properties that specify some time duration should be configured with a unit of time.
+The following format is accepted:
+
+ 5s (seconds)
+ 25ms (milliseconds)
+ 10m or 10min (minutes)
+ 3h (hours)
+ 5d (days)
+ 1y (years)
+
+If no units are provided, the default unit specified in the documentation is assumed.
--- End diff --
[Edit] Actually, scratch my last comment. This is probably fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90716018
> For instance, spark.storage.blockManagerSlaveTimeoutMs used to take in the string 5000, but now I have to pass it 5s otherwise it will throw an exception.
I don't believe that's true; unless the code changed since I last read it, setting it to `5000` should yield the correct result too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28260825
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -174,6 +174,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
getOption(key).getOrElse(defaultValue)
}
+ /** Get a time parameter as seconds; throws a NoSuchElementException if it's not set. If no
+ * suffix is provided then seconds are assumed.
+ */
+ def getTimeAsSec(key: String): Long = {
+ Utils.timeStringAsSec(get(key))
--- End diff --
I personally still prefer these to be `*AsSeconds` so it looks a little more complete. Note that this will be new public API in Spark 1.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27440829
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
+ private def timeStringToUs(str: String) : Long = {
+ try {
+ val lower = str.toLowerCase.trim()
+ if (lower.endsWith("ms")) {
+ lower.substring(0, lower.length - 2).toLong * 1000
+ } else if (lower.endsWith("us")) {
+ lower.substring(0, lower.length - 2).toLong
+ } else if (lower.endsWith("s")) {
+ lower.substring(0, lower.length - 1).toLong * 1000 * 1000
+ } else {
+ // Invalid suffix, force correct formatting
--- End diff --
Not really "invalid suffix", right? This is just handling the case where a suffix is not expected. If there is an invalid suffix, then an exception will be thrown.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88252563
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29502/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91014695
[Test build #29875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29875/consoleFull) for PR 5236 at commit [`d4efd26`](https://github.com/apache/spark/commit/d4efd269f2a8716a46254b87b17a31eb07ec61b4).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196848
--- Diff: docs/configuration.md ---
@@ -989,7 +1002,7 @@ Apart from these, the following properties are also available, and may be useful
</tr>
<tr>
<td><code>spark.locality.wait</code></td>
- <td>3000</td>
+ <td>3000ms</td>
--- End diff --
Same, there are many parameters following this which are described as "in milliseconds" when they aren't in a particular unit anymore (to the caller). Consider writing values of at least 1000ms in seconds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28212581
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -62,14 +62,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext)
// "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses
// "milliseconds"
- private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000))
-
+ private val slaveTimeoutMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerSlaveTimeoutMs", "120s")
+ private val executorTimeoutMs =
+ sc.conf.getTimeAsSec("spark.network.timeout", s"${slaveTimeoutMs}ms") * 1000
+
// "spark.network.timeoutInterval" uses "seconds", while
// "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds"
- private val checkTimeoutIntervalMs =
- sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000).
- getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000))
+ private val timeoutIntervalMs =
+ sc.conf.getTimeAsMs("spark.storage.blockManagerTimeoutIntervalMs", "60s")
+ private val checkTimeoutIntervalMs =
+ sc.conf.getTimeAsSec("spark.network.timeoutInterval", s"${timeoutIntervalMs}ms") * 1000
--- End diff --
Same - it can't since default unit is s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE][WIP] Use consistent naming ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87095158
[Test build #29325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29325/consoleFull) for PR 5236 at commit [`59bf9e1`](https://github.com/apache/spark/commit/59bf9e1d590e49f451aaa7feeb5165572f205358).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90654211
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29797/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792260
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +613,9 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs = Utils.timeStringAsMs(conf.get("spark.files.fetchTimeout", "60s")).toInt
--- End diff --
Just a question -- do you support "1m"? I can't decide whether that's better than "60s". Kind of reads like "1 megabyte" or something unless it's "1min". Either way seems OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-90198164
[Test build #29748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29748/consoleFull) for PR 5236 at commit [`1465390`](https://github.com/apache/spark/commit/14653907e118ead01ec3b0fe6640b5fcaba6d199).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87798995
[Test build #29417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29417/consoleFull) for PR 5236 at commit [`70ac213`](https://github.com/apache/spark/commit/70ac2138d8c1c23bc3bbd4fa91607686b2c329cc).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88138049
[Test build #29485 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29485/consoleFull) for PR 5236 at commit [`bac9edf`](https://github.com/apache/spark/commit/bac9edfe3feb285dc33b2212534841cfe859515f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261106
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -78,8 +78,10 @@ private[spark] object AkkaUtils extends Logging {
val logAkkaConfig = if (conf.getBoolean("spark.akka.logAkkaConfig", false)) "on" else "off"
- val akkaHeartBeatPauses = conf.getInt("spark.akka.heartbeat.pauses", 6000)
- val akkaHeartBeatInterval = conf.getInt("spark.akka.heartbeat.interval", 1000)
+ val akkaHeartBeatPausesS = conf.getTimeAsSec("spark.akka.heartbeat.pauses",
+ "6000s")
--- End diff --
can you move this to the line above? I think it fits comfortably
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261526
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +36,64 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
+
+ test("timeConversion") {
+ // Test -1
+ assert(Utils.timeStringAsSec("-1") === -1)
+
+ // Test zero
+ assert(Utils.timeStringAsSec("0") === 0)
+
+ assert(Utils.timeStringAsSec("1") === 1)
+ assert(Utils.timeStringAsSec("1s") === 1)
+ assert(Utils.timeStringAsSec("1000ms") === 1)
+ assert(Utils.timeStringAsSec("1000000us") === 1)
+ assert(Utils.timeStringAsSec("1min") === TimeUnit.MINUTES.toSeconds(1))
+ assert(Utils.timeStringAsSec("1h") === TimeUnit.HOURS.toSeconds(1))
+ assert(Utils.timeStringAsSec("1d") === TimeUnit.DAYS.toSeconds(1))
+
+ assert(Utils.timeStringAsMs("1") === 1)
+ assert(Utils.timeStringAsMs("1ms") === 1)
+ assert(Utils.timeStringAsMs("1000us") === 1)
+ assert(Utils.timeStringAsMs("1s") === TimeUnit.SECONDS.toMillis(1))
+ assert(Utils.timeStringAsMs("1min") === TimeUnit.MINUTES.toMillis(1))
+ assert(Utils.timeStringAsMs("1h") === TimeUnit.HOURS.toMillis(1))
+ assert(Utils.timeStringAsMs("1d") === TimeUnit.DAYS.toMillis(1))
+
+ // Test invalid strings
+ try {
+ Utils.timeStringAsMs("This breaks 600s")
+ assert(false) // We should never reach this
+ } catch {
+ case e: NumberFormatException => assert(true)
+ }
--- End diff --
there's this really cool thing called `intercept[NumberFormatException]`!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441102
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
--- End diff --
This is a `RuntimeException` so in general there's no need to explicitly declare it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28287786
--- Diff: docs/running-on-yarn.md ---
@@ -48,9 +48,9 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.am.waitTime</code></td>
- <td>100000</td>
+ <td>100s</td>
--- End diff --
unfortunately this needs to be `100000ms` for backward compatibility, because we stated in the docs that these configs accept raw numeric values assuming the units of the default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-89922457
[Test build #29739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29739/consoleFull) for PR 5236 at commit [`5193d5f`](https://github.com/apache/spark/commit/5193d5fe6d36a4aac8fe8c2892177dd216b775ab).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88367541
[Test build #29524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29524/consoleFull) for PR 5236 at commit [`76cfa27`](https://github.com/apache/spark/commit/76cfa2799ceae93534710c28545465c0a2b970f2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441173
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
--- End diff --
nit: I'd declare this where it's used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88308339
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28261462
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -35,7 +36,64 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
class UtilsSuite extends FunSuite with ResetSystemProperties {
+
+ test("timeConversion") {
+ // Test -1
+ assert(Utils.timeStringAsSec("-1") === -1)
--- End diff --
do we actually want this behavior? I would think that if the user tries to give a negative value to a time config we should fail loudly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28212639
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +613,10 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs =
+ conf.getTimeAsSec("spark.files.fetchTimeout", "60s").toInt * 1000
--- End diff --
Can't to maintain backwards compatibility for no units.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88230985
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29494/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88251753
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196805
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -611,9 +613,10 @@ private[spark] object Utils extends Logging {
}
Utils.setupSecureURLConnection(uc, securityMgr)
- val timeout = conf.getInt("spark.files.fetchTimeout", 60) * 1000
- uc.setConnectTimeout(timeout)
- uc.setReadTimeout(timeout)
+ val timeoutMs =
+ conf.getTimeAsSec("spark.files.fetchTimeout", "60s").toInt * 1000
--- End diff --
Same, can go straight to MS?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92493058
[Test build #30184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30184/consoleFull) for PR 5236 at commit [`de3bff9`](https://github.com/apache/spark/commit/de3bff9995f108cdd6183a4fe36b4c5781d861c2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **removes the following dependencies:**
* `RoaringBitmap-0.4.5.jar`
* `activation-1.1.jar`
* `akka-actor_2.10-2.3.4-spark.jar`
* `akka-remote_2.10-2.3.4-spark.jar`
* `akka-slf4j_2.10-2.3.4-spark.jar`
* `aopalliance-1.0.jar`
* `arpack_combined_all-0.1.jar`
* `avro-1.7.7.jar`
* `breeze-macros_2.10-0.11.2.jar`
* `breeze_2.10-0.11.2.jar`
* `chill-java-0.5.0.jar`
* `chill_2.10-0.5.0.jar`
* `commons-beanutils-1.7.0.jar`
* `commons-beanutils-core-1.8.0.jar`
* `commons-cli-1.2.jar`
* `commons-codec-1.10.jar`
* `commons-collections-3.2.1.jar`
* `commons-compress-1.4.1.jar`
* `commons-configuration-1.6.jar`
* `commons-digester-1.8.jar`
* `commons-httpclient-3.1.jar`
* `commons-io-2.1.jar`
* `commons-lang-2.5.jar`
* `commons-lang3-3.3.2.jar`
* `commons-math-2.1.jar`
* `commons-math3-3.1.1.jar`
* `commons-net-2.2.jar`
* `compress-lzf-1.0.0.jar`
* `config-1.2.1.jar`
* `core-1.1.2.jar`
* `curator-client-2.4.0.jar`
* `curator-framework-2.4.0.jar`
* `curator-recipes-2.4.0.jar`
* `gmbal-api-only-3.0.0-b023.jar`
* `grizzly-framework-2.1.2.jar`
* `grizzly-http-2.1.2.jar`
* `grizzly-http-server-2.1.2.jar`
* `grizzly-http-servlet-2.1.2.jar`
* `grizzly-rcm-2.1.2.jar`
* `groovy-all-2.3.7.jar`
* `guava-14.0.1.jar`
* `guice-3.0.jar`
* `hadoop-annotations-2.2.0.jar`
* `hadoop-auth-2.2.0.jar`
* `hadoop-client-2.2.0.jar`
* `hadoop-common-2.2.0.jar`
* `hadoop-hdfs-2.2.0.jar`
* `hadoop-mapreduce-client-app-2.2.0.jar`
* `hadoop-mapreduce-client-common-2.2.0.jar`
* `hadoop-mapreduce-client-core-2.2.0.jar`
* `hadoop-mapreduce-client-jobclient-2.2.0.jar`
* `hadoop-mapreduce-client-shuffle-2.2.0.jar`
* `hadoop-yarn-api-2.2.0.jar`
* `hadoop-yarn-client-2.2.0.jar`
* `hadoop-yarn-common-2.2.0.jar`
* `hadoop-yarn-server-common-2.2.0.jar`
* `ivy-2.4.0.jar`
* `jackson-annotations-2.4.0.jar`
* `jackson-core-2.4.4.jar`
* `jackson-core-asl-1.8.8.jar`
* `jackson-databind-2.4.4.jar`
* `jackson-jaxrs-1.8.8.jar`
* `jackson-mapper-asl-1.8.8.jar`
* `jackson-module-scala_2.10-2.4.4.jar`
* `jackson-xc-1.8.8.jar`
* `jansi-1.4.jar`
* `javax.inject-1.jar`
* `javax.servlet-3.0.0.v201112011016.jar`
* `javax.servlet-3.1.jar`
* `javax.servlet-api-3.0.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
* `jcl-over-slf4j-1.7.10.jar`
* `jersey-client-1.9.jar`
* `jersey-core-1.9.jar`
* `jersey-grizzly2-1.9.jar`
* `jersey-guice-1.9.jar`
* `jersey-json-1.9.jar`
* `jersey-server-1.9.jar`
* `jersey-test-framework-core-1.9.jar`
* `jersey-test-framework-grizzly2-1.9.jar`
* `jets3t-0.7.1.jar`
* `jettison-1.1.jar`
* `jetty-util-6.1.26.jar`
* `jline-0.9.94.jar`
* `jline-2.10.4.jar`
* `jodd-core-3.6.3.jar`
* `json4s-ast_2.10-3.2.10.jar`
* `json4s-core_2.10-3.2.10.jar`
* `json4s-jackson_2.10-3.2.10.jar`
* `jsr305-1.3.9.jar`
* `jtransforms-2.4.0.jar`
* `jul-to-slf4j-1.7.10.jar`
* `kryo-2.21.jar`
* `log4j-1.2.17.jar`
* `lz4-1.2.0.jar`
* `management-api-3.0.0-b012.jar`
* `mesos-0.21.0-shaded-protobuf.jar`
* `metrics-core-3.1.0.jar`
* `metrics-graphite-3.1.0.jar`
* `metrics-json-3.1.0.jar`
* `metrics-jvm-3.1.0.jar`
* `minlog-1.2.jar`
* `netty-3.8.0.Final.jar`
* `netty-all-4.0.23.Final.jar`
* `objenesis-1.2.jar`
* `opencsv-2.3.jar`
* `oro-2.0.8.jar`
* `paranamer-2.6.jar`
* `parquet-column-1.6.0rc3.jar`
* `parquet-common-1.6.0rc3.jar`
* `parquet-encoding-1.6.0rc3.jar`
* `parquet-format-2.2.0-rc1.jar`
* `parquet-generator-1.6.0rc3.jar`
* `parquet-hadoop-1.6.0rc3.jar`
* `parquet-jackson-1.6.0rc3.jar`
* `protobuf-java-2.4.1.jar`
* `protobuf-java-2.5.0-spark.jar`
* `py4j-0.8.2.1.jar`
* `pyrolite-2.0.1.jar`
* `quasiquotes_2.10-2.0.1.jar`
* `reflectasm-1.07-shaded.jar`
* `scala-compiler-2.10.4.jar`
* `scala-library-2.10.4.jar`
* `scala-reflect-2.10.4.jar`
* `scalap-2.10.4.jar`
* `scalatest_2.10-2.2.1.jar`
* `slf4j-api-1.7.10.jar`
* `slf4j-log4j12-1.7.10.jar`
* `snappy-java-1.1.1.6.jar`
* `spark-bagel_2.10-1.4.0-SNAPSHOT.jar`
* `spark-catalyst_2.10-1.4.0-SNAPSHOT.jar`
* `spark-core_2.10-1.4.0-SNAPSHOT.jar`
* `spark-graphx_2.10-1.4.0-SNAPSHOT.jar`
* `spark-launcher_2.10-1.4.0-SNAPSHOT.jar`
* `spark-mllib_2.10-1.4.0-SNAPSHOT.jar`
* `spark-network-common_2.10-1.4.0-SNAPSHOT.jar`
* `spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar`
* `spark-repl_2.10-1.4.0-SNAPSHOT.jar`
* `spark-sql_2.10-1.4.0-SNAPSHOT.jar`
* `spark-streaming_2.10-1.4.0-SNAPSHOT.jar`
* `spire-macros_2.10-0.7.4.jar`
* `spire_2.10-0.7.4.jar`
* `stax-api-1.0.1.jar`
* `stream-2.7.0.jar`
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
* `uncommons-maths-1.2.2a.jar`
* `unused-1.0.0.jar`
* `xmlenc-0.52.jar`
* `xz-1.0.jar`
* `zookeeper-3.4.5.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88253270
[Test build #29503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29503/consoleFull) for PR 5236 at commit [`39164f9`](https://github.com/apache/spark/commit/39164f984800e7474215b1803b594c12def13ae6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88205112
[Test build #29494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29494/consoleFull) for PR 5236 at commit [`3b126e1`](https://github.com/apache/spark/commit/3b126e1cd0da475f1c4c284e656d2125356244a5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88292839
[Test build #29510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29510/consoleFull) for PR 5236 at commit [`bf779b0`](https://github.com/apache/spark/commit/bf779b0e348e599e6b98ceb267a4b30a7ee8ed5a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27792115
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -25,7 +25,7 @@ import org.apache.spark.executor.TaskMetrics
import org.apache.spark.rpc.{ThreadSafeRpcEndpoint, RpcEnv, RpcCallContext}
import org.apache.spark.storage.BlockManagerId
import org.apache.spark.scheduler.{SlaveLost, TaskScheduler}
-import org.apache.spark.util.Utils
+import org.apache.spark.util.{Utils}
--- End diff --
Trivial, but you don't need this change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88304645
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29510/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196917
--- Diff: streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -131,11 +131,11 @@ class ReceiverSuite extends TestSuiteBase with Timeouts with Serializable {
test("block generator") {
val blockGeneratorListener = new FakeBlockGeneratorListener
- val blockInterval = 200
- val conf = new SparkConf().set("spark.streaming.blockInterval", blockInterval.toString)
+ val blockIntervalMs = 200
+ val conf = new SparkConf().set("spark.streaming.blockInterval", s"${blockIntervalMs.toString}ms")
--- End diff --
Tiny: does this need `toString`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87395279
Yeah, the issue is changing the property names, not adding to the values they can take on. I think that's a decent idea myself, to be able to specify "100s" instead of "100000" for a millisecond property.
It raises interesting questions. Is it weird to see `foo.bar.timeoutMS = 100s` ? Of course, that's the reason for changing the property name, but then that itself is disruptive. Also, now you could specify `foo.bar.intervalSec = 1ms` whereas it would not have been possible before to specify an interval below 1 second. I suspect that will be OK, with code changes, but bears verifying.
I swear there was another PR for just this, but I can't find it for the life of me.
This is one where I'd appeal to others for an opinion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87123344
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29331/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91039588
@vanzin - Fixed the potential NPE. Sorry, didn't know about that behavior of containsKey.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-91085117
[Test build #29913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29913/consoleFull) for PR 5236 at commit [`8927e66`](https://github.com/apache/spark/commit/8927e66945983081eef0bf9f04f2ff5eac7bc20d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196572
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -174,6 +174,35 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
getOption(key).getOrElse(defaultValue)
}
+ /** Get a time parameter as seconds; throws a NoSuchElementException if it's not set. If no
+ * suffix is provided then seconds are assumed.
+ */
+ def getTimeAsSec(key: String): Long = {
+ Utils.timeStringAsSec(getOption(key).getOrElse(throw new NoSuchElementException(key)))
+ }
+
+ /** Get a time parameter as seconds, falling back to a default if not set. If no
+ * suffix is provided then seconds are assumed.
+ */
+ def getTimeAsSec(key: String, defaultValue: String): Long = {
+ Utils.timeStringAsSec(getOption(key).getOrElse(defaultValue))
--- End diff --
Same question here, can this be `get(key, defaultValue)` inside?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88181680
[Test build #29490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29490/consoleFull) for PR 5236 at commit [`1858197`](https://github.com/apache/spark/commit/1858197cbb7f0fb75337d25092a525a2aa70f785).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r28196827
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -17,6 +17,9 @@
package org.apache.spark.util
+import java.lang.NumberFormatException
--- End diff --
Trivial: you don't need to import anything from `java.lang`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92453148
[Test build #30176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30176/consoleFull) for PR 5236 at commit [`951ca2d`](https://github.com/apache/spark/commit/951ca2d150e2d0b046573d65bebcb0e15e158c36).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87768485
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29411/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27343096
--- Diff: docs/configuration.md ---
@@ -209,10 +211,10 @@ Apart from these, the following properties are also available, and may be useful
<td>(none)</td>
<td>
Set a special library path to use when launching the driver JVM.
-
+
--- End diff --
Not a big deal but I see a fair bit of white-space-only change in the diff. It makes it a little harder to see what's changed. I don't know how much it's possible or worth fixing by hand but might check your IDE settings and not strip trailing spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87118827
[Test build #29331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29331/consoleFull) for PR 5236 at commit [`9a29d8d`](https://github.com/apache/spark/commit/9a29d8d3f502f43c237859e9f75f5d6c76662915).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5236#discussion_r27441550
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1010,6 +1010,65 @@ private[spark] object Utils extends Logging {
)
}
+ /** Check whether a time-suffix was provided for the time string. */
+ private def hasTimeSuffix(str: String) : Boolean = {
+ val lower = str.toLowerCase.trim()
+ lower.endsWith("ms") || lower.endsWith("us") || lower.endsWith("s")
+ }
+
+ val timeError = "Time must be specified as seconds (s), " +
+ "milliseconds (ms), or microseconds (us) e.g. 50s, 100ms, or 250us."
+
+ /**
+ * Convert a passed time string (e.g. 50s, 100ms, or 250us) to a microsecond count for
+ * internal use. If no suffix is provided a direct conversion is attempted.
+ */
+ @throws(classOf[NumberFormatException])
+ private def timeStringToUs(str: String) : Long = {
+ try {
+ val lower = str.toLowerCase.trim()
+ if (lower.endsWith("ms")) {
--- End diff --
Hmm, you could also use `java.util.concurrent.TimeUnit` here:
val suffixes = Map(
"us" -> TimeUnit.MICROSECONDS,
"ms" -> TimeUnit.MILLISECONDS,
// etc
val us = suffixes(suffix).toMicros(value)
You could then use the same map in other conversions if you really want to:
def timeStrAsHours(value: String) = TimeUnit.MICROSECONDS.toHours(timStringToUs(value))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-92215227
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30138/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87180253
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29339/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87457182
Sean - there's actually only one variable that is still labeled as Ms. All
the other ones don't include units in the parameter name.
On Sun, Mar 29, 2015 at 4:11 AM Sean Owen <no...@github.com> wrote:
> Yeah, the issue is changing the property names, not adding to the values
> they can take on. I think that's a decent idea myself, to be able to
> specify "100s" instead of "100000" for a millisecond property.
>
> It raises interesting questions. Is it weird to see foo.bar.timeoutMS =
> 100s ? Of course, that's the reason for changing the property name, but
> then that itself is disruptive. Also, now you could specify foo.bar.intervalSec
> = 1ms whereas it would not have been possible before to specify an
> interval below 1 second. I suspect that will be OK, with code changes, but
> bears verifying.
>
> I swear there was another PR for just this, but I can't find it for the
> life of me.
>
> This is one where I'd appeal to others for an opinion.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5236#issuecomment-87395279>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87979588
@vanzin I've updated the Scala to use a map like you suggested, still need to update the Java side to do the same if this works. Does this look reasonable? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-88184195
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29485/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5931][CORE] Use consistent naming for t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5236#issuecomment-87112389
[Test build #29329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29329/consoleFull) for PR 5236 at commit [`cde9bff`](https://github.com/apache/spark/commit/cde9bffef6ded083a9dfa80486687ddd07e85640).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org