[jira] [Commented] (SPARK-17339) Fix SparkR tests on Windows

["Shivaram Venkataraman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15464144#comment-15464144 ] 

Shivaram Venkataraman commented on SPARK-17339:
-----------------------------------------------

[~felixcheung] [~hyukjin.kwon] I looked into this a little bit more today and I think the failures are related to SPARK-11227 (or rather the PR at https://github.com/apache/spark/pull/13738/files) -- Specifically the line which calls `FileSystem.get(new URI(path), hadoopConfiguration)` in `hadoopFile` leads to failures with a stack trace that looks like.  
{code}
  java.net.URISyntaxException: Illegal character in opaque part at index 2: C:\Users\shivaram\Downloads\spark-1-sparkr-win-tests-fix\spark-1-sparkr-win-tests-fix\README.md
        at java.net.URI$Parser.fail(URI.java:2829)
        at java.net.URI$Parser.checkChars(URI.java:3002)
        at java.net.URI$Parser.parse(URI.java:3039)
        at java.net.URI.<init>(URI.java:595)
        at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:995)
        at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:990)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.SparkContext.withScope(SparkContext.scala:686)
        at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:990)
        at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:806)
{code}

If I comment out the changes from that PR, the SparkR tests seem to pass. 

[~sarutak] Do you know if there is a better way to fix the issue in SPARK-11227 without trying to construct a URI from the path ?

> Fix SparkR tests on Windows
> ---------------------------
>
>                 Key: SPARK-17339
>                 URL: https://issues.apache.org/jira/browse/SPARK-17339
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR, Tests
>            Reporter: Shivaram Venkataraman
>
> A number of SparkR tests are current failing when run on Windows as discussed in https://github.com/apache/spark/pull/14743
> The list of tests that fail right now is at https://gist.github.com/shivaram/7693df7bd54dc81e2e7d1ce296c41134
> A full log from a build and test on AppVeyor is at https://ci.appveyor.com/project/HyukjinKwon/spark/build/46-test123



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org