You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by taoli91 <gi...@git.apache.org> on 2016/04/26 10:00:50 UTC
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
GitHub user taoli91 opened a pull request:
https://github.com/apache/spark/pull/12695
[SPARK-14914] Normalize Paths/URIs for windows.
## What changes were proposed in this pull request?
The drive label and the back slash on windows' absolute path would confuse the URI scheme. This pull request mostly fix the path issue for windows.
## How was this patch tested?
Unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/taoli91/spark fix_file_path
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12695.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12695
----
commit 2373b56609924b069ec0cfaf10ad887246c0460c
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T03:37:06Z
mvn.cmd for windows
Helping script for windows to download dependency and start zinc to
support incremental building on windows.
commit cc7d9a778d82a5156d7716490ebd8dc405f3791e
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T06:46:32Z
Add helper methods on Utils and Fix fileSuite
commit 9419c36ed3d42ce0e2b3154ebdfe0c68eaef1b06
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T06:56:14Z
Fix the addFile in SparkContext for win path.
When we pass absolute local path with drive label on windows
to SparkContext's addFile, the scheme can't be correctly handled
and it will cause a failure
commit 13559bd80200b81671df77868390f97328050937
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T07:12:34Z
Fix TestUtils and RPackageUtils, still the back slash problem
commit 071fe90f5b7def2083b99500e4d4d9ed7edfa138
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T07:30:27Z
Fix file path in JavaAPISuite
commit 54c720fc3404419019d71c386025230309a7361f
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T07:35:22Z
Fix IvyTstUTils
commit da7d9babc1ba792b192db4eda44b447d5d425e36
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T07:52:17Z
Fix file uri
commit 66b7b4e38475dc7f6acab6517a8cb4878945b0c3
Author: Tao LI <tl...@microsoft.com>
Date: 2016-04-25T07:56:07Z
Fix file path problem for EventLoggingLIstenerSuite
commit 226c2d489a8b7cb3a18b0533023462a010e68a4a
Author: Tao LI <tl...@microsoft.com>
Date: 2016-04-25T08:05:48Z
Fix the file paths.
commit f07df07dd26b3b7b8cc3bac4221ad28cda852bd7
Author: Tao LI <tl...@microsoft.com>
Date: 2016-04-25T08:08:35Z
Fix the file paths.
commit 54a0f27d557b6f1b4a6157a16bf65de37944d327
Author: Tao LI <tl...@microsoft.com>
Date: 2016-04-25T08:23:40Z
Slash problem
commit 973564ae6c7475e354a1b6d2d7b5a54d716cd2e3
Author: Tao LI <tl...@microsoft.com>
Date: 2016-04-25T08:30:52Z
Another fix.
commit 9dad154905ea6cdc0d77966c4475eee0bd8b1da2
Author: U-FAREAST\tl <tl...@microsoft.com>
Date: 2016-04-25T13:47:56Z
Fixed file path
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12695#discussion_r61046782
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -416,9 +416,9 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
assertResolves("hdfs:/jar1,file:/jar2,jar3,jar4#jar5,path to/jar6",
s"hdfs:/jar1,file:/jar2,file:$cwd/jar3,file:$cwd/jar4#jar5,file:$cwd/path%20to/jar6")
if (Utils.isWindows) {
+ // It seems "#" is not valid for a local file path, and it will be escaped by the File.toURI
assertResolves("""hdfs:/jar1,file:/jar2,jar3,C:\pi.py#py.pi,C:\path to\jar4""",
- s"hdfs:/jar1,file:/jar2,file:$cwd/jar3,file:/C:/pi.py#py.pi,file:/C:/path%20to/jar4")
- }
+ s"hdfs:/jar1,file:/jar2,file:$cwd/jar3,file:/C:/pi.py%23py.pi,file:/C:/path%20to/jar4") }
--- End diff --
Ditto.
```
s"hdfs:/jar1,file:/jar2,file:$cwd/jar3,file:/C:/pi.py%23py.pi,file:/C:/path%20to/jar4")
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12695#discussion_r61046667
--- Diff: core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendSuite.scala ---
@@ -25,20 +26,18 @@ import java.util.Collections
import scala.collection.JavaConverters._
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
-
import org.apache.mesos.{Protos, Scheduler, SchedulerDriver}
import org.apache.mesos.Protos._
import org.apache.mesos.Protos.Value.Scalar
import org.mockito.{ArgumentCaptor, Matchers}
import org.mockito.Matchers._
import org.mockito.Mockito._
import org.scalatest.mock.MockitoSugar
-
import org.apache.spark.{LocalSparkContext, SparkConf, SparkContext, SparkFunSuite}
import org.apache.spark.executor.MesosExecutorBackend
-import org.apache.spark.scheduler.{LiveListenerBus, SparkListenerExecutorAdded,
- TaskDescription, TaskSchedulerImpl, WorkerOffer}
--- End diff --
Please do not remove newlines between imports, (See [databricks/scala-style-guide#imports] (https://github.com/databricks/scala-style-guide#imports))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214672512
@srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-215282915
@HyukjinKwon Thanks for pointing out, I haven't noticed that there is an auto style check. Im going to have a vacation for a few days and I will fix them as soon as possible next week
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12695
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-215425968
Yes, I'm assuming all of the inputs need to be parseable as a URI in order to keep this sensible. At least, that seems less problematic than trying to accept inputs like `D:\foo\bar`. `/foo/bar` is ambiguous since it could be a local or HDFS file, in which case it needs a scheme. This does probably mean some pieces of the code which deal with local files via string paths need some extra logic to parse as a URI and get the parsed path out of it. I think that's substantially what you're doing here, just wondering if we can make it simpler by making stronger input assumptions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214662771
@taoli91 please combine all your PRs into one and close the others.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214663838
@srowen Sure, I'll combine them into one. Could you elaborate more about what logic am I re-implementing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-215369309
Isn't the URI just `file:///D:/test`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/12695
As #13868 does adopt `org.apache.hadoop.io.Path`, I don't see this patch being needed \u2014though it may highlight some places where the new code may need applying
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-215403420
@srowen `new FIle("D:\test").toURI.toString` can return `file://D/test` but `new URI("D:\test")` won't. But we should make sure that it is a local file before converting it into a file. Actually lots of method accepts a path string, and it will be converted to URI internally. Most test cases pass in path string instead of URI like string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12695#discussion_r61046496
--- Diff: core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala ---
@@ -61,8 +61,7 @@ class InputOutputMetricsSuite extends SparkFunSuite with SharedSparkContext
pw.close()
// Path to tmpFile
- tmpFilePath = "file://" + tmpFile.getAbsolutePath
- }
+ tmpFilePath = Utils.resolveURI(tmpFile.getAbsolutePath).toString }
--- End diff --
Maybe,
```scala
tmpFilePath = Utils.resolveURI(tmpFile.getAbsolutePath).toString
}
```
(See [databricks/scala-style-guide#imports](https://github.com/databricks/scala-style-guide#curly-braces)).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214672918
> please combine all your PRs into one and close the others.
@srowen Double think about this problem, it seems the PR is too large if combining into one. I personally believe that it's better to categorize the changes and separate them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/12695
Let's close this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the issue:
https://github.com/apache/spark/pull/12695
@vanzin This PR partly addresses a broader problem that path construction is not consistent throughout Spark sources. Indeed, this is a potential source of bugs. Here is an example https://github.com/apache/spark/pull/13868#issuecomment-228932793. However, author addresses the problem with regex, which does not seem to be optimal as @srowen mentioned. I believe we should have a separate JIRA for making all path references and construction error-prone and consistent, and check everything including shell scripts. For example, https://github.com/apache/spark/blob/master/build/mvn#L161 contains a reference to `${MVN_BIN} -someParameters` which will fail if maven path contains a space.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214668100
Broadly: you should be able to use the `File` API with forward-slash paths to describe and manipulate file locations, in a platform-independent way. Wherever you're changing Strings to Files is probably a good thing. Where you're manually handling paths with regexes seems suboptimal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214673990
The problem is that discussion gets split across many threads, when the changes have a logical relationship. I see that they're mostly distinct threads of change in this case. Some of these may end up not being merged or in reduced form, which makes them more combine-able. For now, leave them to see where they go
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214662914
Most of this PR is not the way to do it and adds lots of complexity. You're manually reimplementing a lot of logic in the JDK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/12695
Is this needed after #13868? pinging @avulanov
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214661123
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12695#discussion_r61045917
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1313,11 +1313,8 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
* supported for Hadoop-supported filesystems.
*/
def addFile(path: String, recursive: Boolean): Unit = {
- val uri = new URI(path)
- val schemeCorrectedPath = uri.getScheme match {
- case null | "local" => new File(path).getCanonicalFile.toURI.toString
- case _ => path
- }
+ val uri =Utils.resolveURI(path)
--- End diff --
Maybe `val uri = Utils`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/12695
If you are working with windows paths; Hadoop's Path class contains the code to do this, stabilised and addressing the corner cases
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-214676529
(@taoli91 FYI, it would be great if you run `sbt scalastyle` or `./dev/run-tests` (this triggers style checking first) for style corrections before adding some commits. This will show up the corrections I said in the comments.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by taoli91 <gi...@git.apache.org>.
Github user taoli91 commented on the pull request:
https://github.com/apache/spark/pull/12695#issuecomment-215283532
> Where you're manually handling paths with regexes seems suboptimal.
@srowen The regex parts are actually dealing with the URI. The default URI can't correctly handle the absolute Windows local path. It will recognize the drive label to be scheme. For example `new URI("D:\test")` will return a scheme of D. I can't find a JDK method to handle this problem, thus I have to use regex to find out the label and manually append a `/` in front of the path. E.g., convert `D:\test` to `/D:/test` before passing to the constructor of URI
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14914] Normalize Paths/URIs for windows...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12695#discussion_r61046074
--- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java ---
@@ -35,6 +35,7 @@
import java.util.Set;
import java.util.concurrent.*;
+import org.apache.spark.util.Utils;
--- End diff --
Please see [databricks/scala-style-guide#imports](https://github.com/databricks/scala-style-guide#imports).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org